Well, the job is interesting, I spend lot of dollars on GCP =D. But if I would have to choose, without money been an issue, I would go back to the academy.
Just so you don't get the impression every physicist is having a fabulous time, here's me:
I have a bachelors in physics. I went for a phd in biology but had to bail to work to support family, plus I can't seem to come up with original ideas.
I now have a job with physicist as part of the title, but what I really do is try to read bad handwriting from records from nuclear weapons plants. It's a dull job a monkey could do but it pays the bills. For years I worked as a computer guy for an academic department and that was fun and I'm trying to get back into it but nothing yet. I teach math at a community college for fun too.
I have great kids, so all and all I'm happy about things, but I am sad the physics thing never panned out. As an undergrad I was excited about chaos and nonlinear dynamics. Still read math texts for fun and play with Haskell.
To be entirely fair, people not educated in physics would likely have a very hard time at your job because we don't know the terminology or what makes sense for your field. If I tried to do what you do, I'd likely have no clue what half of the pigeon-scratch is because I flat out do not even know what the words or concepts are.
Still, I feel I could take any of my students and in one month have them up to speed on what to look for and what things mean.
And I don't mean to suggest that finding challenging work is any sort of failing of my employer. I have been able to diversify the position to include doing graphics in R and some stats things and, along with some other people, I think we can sell some sexy Bayesian things.
They are very receptive of new ideas, but... it is not a growing industry (thankfully), so any sort of advance is going to be a jump to a different field. Which is possible, and I'm working on it.
My point primarily is when you read these things about people doing all sorts of wonderful exciting things, you get the impression that what you have done or are doing is crap. Perhaps it's nice to hear that a lot of people do not set the world on fire. Takes a little pressure off.
I think you could swing some academic or national lab position if you put some time into it.
Is there anything even tangentially related to your dayjob that you could mine for ideas? Or maybe take some time to come up with some research direction?
A places like Oak Ridge could have something interesting available.
Devaney was my advisor's advisor when he did he his PhD at BU. I took a course on discrete dynamics and chaos and wrote my senior year thesis on bifurcation diagrams.
I had his Chaotic Dynamical Systems book checked out from the library for years and carried it everywhere. I should find a copy, what a great book!
I wonder if there's a good sort of "postmortem" on what happened to NLD and Chaos. It seems like everyone said, "Ok, that's a thing and it explains a lot but we can't do anything with it."
As a further aside, I do get to do some fun graphics in R and I'm pretty impressed how R has become the statistical standard and graphics too, mainly I think from one super dynamic statistician.
Now that R is entrenched I can work on moving everyone into Haskell!
So pure and innocent. I've found that programmers are not too fond of having even a simple IO operation lead to discussions of category theory. Good luck in your noble lambda crusade :-). The future is functional! Hack the type systems!
Would you have any interest in doing Haskell full-time? I know several well-funded companies looking for folks with similar backgrounds to yours, in SF, NYC, and London. Happy to connect you if you’re looking for something more interesting and possibly better pay.
Sure, the two I had in mind in NYC are Kadena (http://kadena.io) and Jane St. (https://www.janestreet.com). Jane St. is more an OCaml shop but they like Haskell people. Kadena builds everything in Haskell.
"""try to read bad handwriting from records from nuclear weapons plants. It's a dull job a monkey could do but it pays the bills."""
+
"""For years I worked as a computer guy for an academic department and that was fun and I'm trying to get back into it"""
Sounds like you could enjoy automating your work. Learn a bit of Python and some Machine Learning/Deep Learning and digitize the handwriting and build a little program that reads the stuff for you.
The default example for reading handwritten digits is called MNIST if you want to read further. I'd suggest fast.ai and watching the first couple of lessons. That should get you started to play around with this. You don't have to tell anyone that you do this (I probably wouldn't) but hey at least it might be a nice way to do a little less boring stuff and ease into a bit of programmin/data science?
First of all, you definitely don't want any chance of incorrect recognition of handwriting coming from a nuclear weapons plant.
Secondly, who knows what policies he might violate by using some unauthorized, untested software (whether OP is the author or not) with potentially sensitive information.
He can use it such that the model's predictions are still manually verified by him before actually submitting them or whatever. Seems pretty harmless to me.
In his current situation, if he zones out, he produces nothing. In the situation where he's aided by the potentially faulty described system, if he zones out, he produces erroneously transcribed data.
I'd first go for a pareto approach. There's probably some easy automation that can aid the manual work, which is easy to implement and provides visible performance gains. Stuff like pre selecting interesting image parts (aim for high false positive rates and zero false negatives) or even just an interface for speeding up dull manual work like pulling documents from the repository, associating relevant documents and presenting them in a fast acting UI.
I've done something similar, automating a similar process. Essentially the screen was split on half. The left was the actual image, the right was the data of what the program thinks it is after OCR and some magic interpretation. The user can scroll around and both panes would move together. User clicks accepts on each value. Otherwise user can edit each value, or add a new value (if it was completely missed) and then accept. Once everything is accepted they can save the data.
On paper it saved a lot of man hours but it was never fully rolled out. The project didn't have a long lifespan and we had enough people sitting around doing nothing to just manually do all the work. But I got a nice award for it LOL.
This is more like some guy in the early 1900s having the job of hand-delivering important information on a horse.
Then deciding on his own on the advice of HN that he should try a car for that task instead, the car breaking down, and the guy getting fired because his employer never approved of this whole "car" thing and he shouldn't have introduced that into his job workflow without having talked to his employer about it.
I will take a look at it, thanks. I have tried some off the shelf OCR things and had little success. But what I do is really needle in the haystack kind of thing. I am looking for a "U" or and "Sr" in a particular place and happy to find one in a thousand pdf pages.
I am looking into auto-scrolling pdfs and batch loading, sequencing of pdfs to make things easier. But I will definitely check out fast.ai - thanks!
Only problem is I don't know how likely it would be that the writing is neatly separated by character for mnist to work, and handwriting recognition isn't accurate enough. Maybe some restraints on the inputs will fix that
Regretfully pragmatic viewpoint here: The poster is currently relying on this menial job for income. Automating it before he has something else lined up could be a really bad idea, financially.
