On the note of Jupyter notebooks and version control - there was a talk at this year's Pycon Ireland about using a built in cleaner for notebooks when committing the JSON (discard the cell results), and then dropping the whole lot into a CI system utilising remote execution (and Bazel or similar) to run and cache the outputs. Was a talk from CodeThink. No video up yet though. Scenario was reproducible notebooks for processing data from a system under test.
> I'd go further and say you're not a senior if your code isn't good you shouldn't be a senior.
You say that until you are tasked with doing impossible - three lines, all perpendicular, five green, two anti-green, seven in ten or more dimensions, any color; while customer only uses purple lines.
Last guy that worked on it committed seppuku. Rest of team is in mental ward. Your only team member is guy that programmed his entire life in PHP, and doesn't know backend's language. Just teach him.
Documentation, is spread between Jira, wiki, Markdown, ftp server and some napkins.
CI stands for continuous Indians. You send code to India, where a team will assemble it. It may take anywhere between a few minutes or few hours. But it beats GitHub actions. Make sure to inspect artifacts, the Indian team has a habit to add some of their ""bug fixes"" covertly.
What an unimaginable horror! You can't change a single line of code in the product without breaking 1000s of existing tests. Generations of programmers have worked on that code under difficult deadlines and filled the code with all kinds of crap.
Very complex pieces of logic, memory management, context switching, etc. are all held together with thousands of flags. The whole code is ridden with mysterious macros that one cannot decipher without picking a notebook and expanding relevant pats of the macros by hand. It can take a day to two days to really understand what a macro does.
Sometimes one needs to understand the values and the effects of 20 different flag to predict how the code would behave in different situations. Sometimes 100s too! I am not exaggerating.
The only reason why this product is still surviving and still works is due to literally millions of tests!
Here is how the life of an Oracle Database developer is:
- Start working on a new bug.
- Spend two weeks trying to understand the 20 different flags that interact in mysterious ways to cause this bag.
- Add one more flag to handle the new special scenario. Add a few more lines of code that checks this flag and works around the problematic situation and avoids the bug.
- Submit the changes to a test farm consisting of about 100 to 200 servers that would compile the code, build a new Oracle DB, and run the millions of tests in a distributed fashion.
- Go home. Come the next day and work on something else. The tests can take 20 hours to 30 hours to complete.
- Go home. Come the next day and check your farm test results. On a good day, there would be about 100 failing tests. On a bad day, there would be about 1000 failing tests. Pick some of these tests randomly and try to understand what went wrong with your assumptions. Maybe there are some 10 more flags to consider to truly understand the nature of the bug.
- Add a few more flags in an attempt to fix the issue. Submit the changes again for testing. Wait another 20 to 30 hours.
- Rinse and repeat for another two weeks until you get the mysterious incantation of the combination of flags right.
- Finally one fine day you would succeed with 0 tests failing.
- Add a hundred more tests for your new change to ensure that the next developer who has the misfortune of touching this new piece of code never ends up breaking your fix.
- Submit the work for one final round of testing. Then submit it for review. The review itself may take another 2 weeks to 2 months. So now move on to the next bug to work on.
- After 2 weeks to 2 months, when everything is complete, the code would be finally merged into the main branch.
The above is a non-exaggerated description of the life of a programmer in Oracle fixing a bug. Now imagine what horror it is going to be to develop a new feature. It takes 6 months to a year (sometimes two years!) to develop a single small feature (say something like adding a new mode of authentication like support for AD authentication).
The fact that this product even works is nothing short of a miracle!
I don't work for Oracle anymore. Will never work for Oracle again!
This is one of those things that you don't really tend to think about (pun not intended!) until you experience a change in your thinking or meet someone who thinks like you do!
> If we can avoid the compression step, and do the manipulations directly in the high-dimensional, non-linguistic, conceptual space, we can move much faster
With my neurodivergent brain I've always conducted my thoughts in an "uncompressed format" and then eternally struggled to confine it all into words. Only then for people to misinterpret and question it. They might get caught up in the first sentence when the end of the paragraph is where you need to be!
That's why when you meet someone who thinks like you the depth of conversation and thinking you can achieve together is vast and also incredibly liberating! Your no longer limited by words in same way.
Since becoming ill I've suffered badly with brainfog. The cutesy name for a cruel experience. Sometimes there's no memories to draw on when your thinking, the cupboards are bare. You can't leap from thought to thought because they disappear before you get there or after like a cursed platformer. You might be able to grab hold of the thought but you can't reach inside or read it. It's all wrong somehow like when your suddenly convinced a word is spelt wrong even though you know it's right. You can't maintain focus long enough to finish your train of thought.
Even that subconscious processing is affected I used to prime my brain with information all day and instead of waking up with the solution I'll wake up frustrated but not knowing why. Just the vague notion that I failed at something that used to come so easily.
I think what happened was that Jim Hugunin wrote Numeric (Numerical Python) in 1995 with help from a bunch of other people, and Travis Oliphant and some other people put together SciPy in 2001 from modules they'd been working on previously. A hecking lot of work had already gone into Numeric and SciPy by the time I first tried using it, around 2003, and so it was already pretty stable, with a broad range of capabilities. It had already left Perl's PDL far behind (except for 3-D plotting support), and Ruby's NArray barely existed, being under 10KLOC when it was imported from CVS in 2001: https://github.com/masa16/narray/commits/master?after=447c96...
So Numeric and then SciPy (2001), numarray (2001), and Numpy (2005) just had a big head start on the other free-software numerical computation libraries, basically because a few early Pythonistas were interested in the field, and it kept accruing more and more powerful stuff: FFTW (1998), SparsePy (1999), the ability to read MATLAB files (1999), matplotlib (2001), IPython (2001), and so on. It got popular enough that dozens of people started contributing code to it, making it grow faster and faster.
At the time, most scientists were still using MATLAB; other alternatives for array-based computing included R, Bell Labs's Lush, and if you were a real masochist, PV-WAVE's IDL. Octave, the free-software MATLAB clone, wasn't yet a reasonable replacement. And all of these languages had the disadvantage that they were sort of trapped in their own little library ecosystems: if you wanted to do Gaussian quadrature or calculate a Kullback-Leibler divergence or plot some data, you were golden, but if you wanted to scrape some data out of some web pages with regular expressions, parse command-line flags, read members of a zip file, or run as a CGI process on a web server, you were shit out of luck. Numpy's syntax for a vector of numbers is pretty shitty compared to Octave or APL, but it's worlds better than R or especially IDL, and most people prefer it to Lush's Lisp syntax, although I have a soft spot for Lush myself. And Python and Numpy have reasonable language semantics too, unlike Octave or R.
(I think Lush may have been a bit late to the free-software party, too, so it wasn't really an option.)
So academia started switching over from MATLAB to Numpy, I think around 2010. The transition isn't over yet; the numerical-methods class I audited last year is still using Octave. R is dominant in statistics and will probably stay that way. But Numpy is pretty broadly adopted.
It basically came down to a head start due to a lot of hard work done in the 1990s that just happened to be done in Python. It's maybe not entirely coincidence that the people whose taste ran to reimplementing Gaussian quadrature were fans of Python and not Perl, Ruby, Lua, or Tcl, but they could have been.
Yes and no, for human search - its kinda neat, you might find some duplicates, or some nearby neighbour bugs that help you solve a whole class of issues.
But the cool kids? They'd do something worse;
They'd define some complicated agentic setup that cloned your code base into containers firewalled off from the world, give prompts like;
Your expert software dev in MY_FAVE_LANG, here's a bug description 'LONG BUG DESCRIPTION' explore the code and write a solution. Here's some tools (read_file, write_file, ETC)
You'd then spawn as many of these as you can, per task, and have them all generate pull requests for the tasks. Review them with an LLM, then manually and accept PR's you wanted. Now your in the ultra money.
You'd use RAG to guide an untuned LLM on your code base for styles and how to write code. You'd write docs like "how to write an API, how to write a DB migration, ETC" and give that as tool to the agents writing the code.
With time and effort, you can write agents to be specific to your code base through fine tuning, but who's got that kind of money?
That's definitely insightful. Everyone reaches a level where coasting on smarts is no longer sufficient.
Many reach this realization when starting university, but some can still coast okay in college since the material to learn is well defined and upper bounded. A PhD is not really upper bounded. There's no set out amount of papers to read per week like in a college course. There's no "this won't be part of the exam". Anything is fair game. The returns on being smarter never flatten out, but simply there's no ceiling. You can always do more, read more to keep up with the literature firehose, improve your experiments, your method, etc.
You also need soft skills and a network. You need to keep your finger on the pulse of the community by going to conferences and getting to know people, grabbing coffee or going out to dinner with them. You also need to be slef driven instead of waiting for instructions like it was in college. You need to be just the right amount of skeptical and critical regarding existing methods to be able to come up with new things while being also understood and accepted and seen relevant and exciting by the community.
You also need to manage your time and set your own deadlines and maintain a routine without the external sync given by university lectures and exams. All this basically has no upper limit and even the expectations are vaguely defined. You face rejections maybe for the first time despite having done a thorough work because the reviewers don't see enough novelty or it doesn't slot neatly into what is in fashion at the moment.
My point is that a PhD can push everyone to meet their mental limits. It can be frustrating and it's a notoriously hard period of time for many PhD students. Of course if your only goal is to graduate to get the doctorate, there are possible strategies to "coast", but those who go for the academic path often expect to achieve more than the bare minimum, especially if they managed to coast with good results in college.
A key aspect of the Carmack approach (or similar 'smart hacker' unconventional career approach) is avoiding that situation in the first place. However, this also carries substantial career, financial and lifestyle risks & trade-offs - especially if you're not both talented enough and lucky enough to hit a sufficiently fertile oppty in the right time window on the first few tries.
Assuming one is willing to accept the risks and has the requisite high-talent plus strong work drive, the Carmack-like career pattern is to devote great care to evaluating and selecting opptys near the edges of newly emerging 'interesting things' which also: coincide with your interests/talents, are still at a point where a small team can plausibly generate meaningful traction, and have plausible potential to grow quickly and get big.
Carmack was fortunate that his strong interest in graphics and games overlapped a time period when Moore's Law was enabling quite capable CPU, RAM and GFX hardware to hit consumer prices. But we shouldn't dismiss Carmack's success as "luck". That kind of luck is an ever-present uncontrolled variable which must be factored into your approach - not ignored. Since Carmack has since shown he can get very interested in a variety of things, I assume he filtered his strong interests to pick the one with the most near-term growth potential which also matched his skills. I suspect the most fortunate "luck" Carmack had wasn't picking game graphics in the early 90s, it was that (for whatever reasons) he wasn't already employed in a more typical "well-paying job with a big, stable company, great benefits and career growth potential" so he was free to find the oppty in the first place.
I had a similarly unconventional career path which, fortunately, turned out very well for me (although not quite at Carmack's scale :-)). The best luck I had actually looked like 'bad luck' to me and everyone else. Due to my inability to succeed in a traditional educational context (and other personal shortcomings), I didn't have a college degree or resume sufficient to get a "good job", so I had little choice but to take the high-risk road and figure out the unconventional approach as best I could - which involved teaching myself, then hiring myself (because no one else would) and then repeatedly failing my way through learning startup entrepreneurship until I got good at it. I think the reality is that few who succeed on the 'unconventional approach' consciously chose that path at the beginning over lower risk, more comfortable alternatives - we simply never had those alternatives to 'bravely' reject in pursuit of our dreams :-).
I think you might be suffering from a categorical blindness to a certain type of thing humans do.
Let's say I own a private beach. I want to allow my beach to be enjoyed freely and responsibly by a reasonable number of people, whether friends or strangers. I don't want to constantly be cleaning up garbage on my beach. And I don't want the beach to be overcrowded when I myself use it.
So what do I do? Well, I'm sure not going to hire a bouncer to guard my beach. (How would I even tell them who's allowed in, anyway? Can you recognize "irresponsible people" on sight?)
No, instead, I will probably post a sign outside my beach, saying "NO TRESPASSING".
But I won't enforce it! And if anyone (e.g. my few direct friends who I invite to hang with me at my beach) asks, I'll tell them I won't enforce it! They can bring people to my beach if they like!
Access to the beach is now an open secret. It's something that people can freely tell those they trust about. The number of people visiting the beach will rise slowly over time. Maybe it'll eventually increase to be too much; or maybe it'll level off, due to churn in the population near the beach. (Mostly depends on how hard the beach is to access, and the demographics that live nearby.)
If some tour company tries to drop off a whole busload of tourists at my beach, though, I will most certainly kick them out, pointing at the "NO TRESPASSING" sign. (Since I don't have a bouncer, probably what I would actually do is call the cops on them.)
The cops would ask me about the people already on the beach, of course. To which I would say:
> Those people on the beach right now? They're my "friends." No, I don't exactly know them... but I know people who know them! They're "on the guest list." But these people standing by the bus over here — these are not my friends. These are people brought here by a guy trying to profit off of providing others access to my beach, which I have not granted. They are not allowed in. Nobody brought here by this bus company will ever be allowed in.
This is every underground party ever. This is every travel destination for the rich. Open secrets, with guardians who actively lie by exaggerating the restrictions or conditions in place, to keep a lid on the spread of the secret.
And this is a thing companies do constantly.
• Every store discount code given out to some YouTuber to give to people who watch their thing? Open secret. (Consider: is it "legitimate" for a discount app like Honey to find and publish those audience-targeted codes? No, probably not; Honey would be acting like the tour-bus operator above. But would the online store mind if you personally found the code and used it, despite not being a member of that Youtuber's audience? No, they'd be happy to have your business. Would they even mind if you told three friends, and you all immediately bought something? No. In fact, they'd be overjoyed!)
• The unmentioned (and implied to the contrary!) never-ending-ness of the free trial period for WinRAR? Open secret. (If WinRAR never implied you had to buy it at some point, nobody would have ever bought it; they'd just consider it freeware. But you don't "have" to buy it. It goes on working forever. Some people feel guilty or pressured, and do buy it. Others eventually discover the bomb is a dud. This is WinRAR's intended business model.)
• The CPU binning lottery? Open secret. (Did you know you can keep RMAing retail-purchased CPUs until you get a really highly overclockable one? You do now! And people have been doing this for decades! CPU vendors don't care—in fact, they want these few super-enthusiasts to get their hands on their best CPUs, since they'll probably publish some really nice benchmarks with them. Free advertising! They certainly don't want a company doing this in bulk though. That'd be way more trouble than it's worth; and then what would they do with a huge pile of RMAed known-below-average-binned CPUs?)
• How easy Photoshop was to pirate in the pre-Creative-Cloud era? Open secret. (See my sibling post.)
You can exploit any/all of these if you know (and you're not in a situation legally preventing you from doing so — e.g. corporations can't pirate things.)
And some people know; but most people don't.
This equilibrium state is exactly the point aimed for by the corporations that create these open secrets. They don't want these secrets known by everyone. (If enough people do it, then it's no longer a marketing expense, but a hole in their business model.) But they don't want these secrets known by nobody, either.
The creators of any open secret, want some deserving people to take advantage of the open secret; otherwise they wouldn't have made it an open secret. (In almost all cases, you have to actually do extra work to make something an open secret. It's extra work to carefully design and manage the "virality coefficient" of an open secret so that it'll hit equilibrium, rather than spreading to fixation or dying out. The outbound word-of-mouth advertising required to get an underground party to happen, for example, is way more work than just putting up posters! It would almost always have been easier to just have no secret at all!)
I hope you will agree with me that this dynamic exists in general.
If you do: what then leads you to believe that what Apple has here is a dumb unenforced mistake, rather than an open secret?
---
One extra point, that doesn't have a clean place to insert above: corporations are really careful with the way they structure the wording of the exaggerated-restriction "wards" shrouding their open secrets.
For a person, a "TRESPASSING A-OK" sign would just be a sign. But for a corporation, any positive criteria they give implying that a group does qualify for a certain promotion, can be taken as a legal promise on their part.
If Apple offered an obscure promotion to "anyone who can find it" — some secondary secret version of their online store that just happens to have lower prices, say — and then some bigcorp found it... and if Apple then attempted to refuse to apply those promotional prices to that bigcorp's 100k-seat volume purchase of Mac Studios or whatever they were trying to get away with — then the bigcorp could actually be in their right to sue Apple for breaking the promise they were making by having such a store available without qualification! (a.k.a. promissory estoppel.)
(To be clear, to win such a case, the bigcorp would have to also prove that they then went out and did something under the assumption that they could get those 100k Mac Studios at that price — bought 100k Mac Studio-shaped desk nooks, say — and that by being refused the promotion, this contingent action has resulted in a financial loss for them — e.g. if it turns out the 100k nooks have zero resale value, so they're out the cost of the nooks, and also have a huge pile of useless plastic it'll probably cost money to dispose of. But that's not too uncommon of a problem to have, in a big-enough corp with many async/concurrent/pipelined corporate purchasing negotiations going on. So it's something the legal departments of vendors like Apple are always wary of accidentally getting tangled up in.)
"Students and teachers" is a particularly nice/"safe" wording for open-secret shrouding language for a corporate promotion, because there is no case in which a corporation qualifies as a student or a teacher. And yet literally anyone else can become a student at any time, just by signing up for a zero-tuition-until-you-take-courses online university program and nabbing the resulting .edu email. (By the premise of continuous education/lifelong learning, we are always students!) "Students and teachers" is a group that any price-conscious motivated individual can join trivially (just like clipping a coupon!), but which keeps the corporate-buyer discount-loophole-hunters out.
The architecture of AeroSpace is that on every user input that may change window configuration (new window created, window moved, window resized, new app launched, etc), AeroSpace runs the same idempotent operation (I call it "refresh session") that tries to detect new window, checks all invariants, re-layouts windows, etc.
The "refresh session" performs all the mentioned steps regardless of the user input nature (it doesn't matter whether the window is moved, or a new app is launched)
I believe that this architecture may lose windows only if the macOS API returns invalid data.
I have been using AeroSpace for quite a while myself and I'm happy with it
I run a content website for a living. This year I switched from Craft CMS to my own static site generator.
I no longer think about the server or the CMS. I no longer need to keep it updated. I got rid of the heavy database and the elaborate caching setup. Now it's just a static file server. If it was just the blog it would be even easier to host. The website is more reliable and requires virtually no maintenance.
The best part for me is that I can work offline. It's just me and my text editor, so even my tiny Macbook 12" feels blazing fast.
Version control is also incredibly valuable. I can review my changes or roll them back. I can also find/replace across the entire content with regex. Text files are easy to work with.
The film was shot and framed in 4:3 but was cropped down (for reasons unknown to me) into widescreen for the theatrical and later the Blu-ray releases.
There is a rare uncropped DVD release of it by a distributor called "IRE" out there. My understanding is that every copy of this release is autographed by the director, if you need some proof of artistic intent. AFAIK there's no definitive proof that this isn't just "open matte", but IMO the superiority of this release is pretty clear. You can find it on public torrent sites.
I tried to create this type of culture at my last gig, where I had the unusual privilege of being able to hire almost the entire engineering team, alongside my manager who was also very document oriented. Unfortunately, it didn't work out. Maybe Tremendous has done tremendously better, it's certainly possible, but here is a list of things that went wrong, maybe it's useful.
1. Standard interviews don't assess reading/typing speeds. If you want a high documentation culture this is critical. It took way too long for us to figure this out but many people in the company were significantly slower at reading/typing than us; they found long documents overwhelming and would find excuses to not read them. Slack conversations became a massive sore spot because unknown to us some people felt like they couldn't keep up. They'd try to type a question or response and we'd already posted another two paragraphs before they got a chance to finish their thought. They'd complain to each other that if they asked us a question they got back an essay in response, etc.
2. Documentation requires ownership, otherwise it rapidly becomes useless. Standard corp tooling like wikis doesn't make such workflows easy. They are however optimal from a corp politics perspective (dispersal of responsibility). Maintaining markdown based websites works well as long as you have empowered maintainers who view document quality as a core job function, but you have to force people to submit changes by e.g. rejecting at code review time changes that don't update the docs. People will moan. They will ask you to do it for them. They will submit absolutely min-viable docs changes, they will demand you hire technical writers even if they're easily capable of doing it themselves. And of course the moment you're not using a git-style workflow, just forget it, you have no chance of preserving coherency in any sort of knowledge base.
3. Lots of people aren't just slow but actively HATE reading and writing. They will make things up on the spot, or lie, or just flat out refuse to do the work rather than sit down and read a long document. Jeff Bezos has said about why Amazon uses meeting time to force people to read the memo:
"If we don’t, the executives, like high school kids, will try to bluff their way through a meeting"
You will have to fire people for refusing to read things if you're serious about creating and maintaining such a docs-oriented culture, which in practice is so unpleasant nobody ever does it and so maintaining such a culture is nearly impossible. You will also have to flat-out refuse to meet people in order to force them to read, because otherwise they'll receive a document and just ignore it. I had several cases where one of my most senior engineers would assert that a product we used didn't have feature X, and I had to correct him by pointing out that the user manual discussed feature X in detail. I knew this because I'd actually read the user manual cover to cover. Basically nobody does this and guess what, if you're the one person on the team who reads stuff then you're going to come across as the awkward smart alec who makes people look stupid. Sometimes, ignorance is bliss.
What I have noticed about western photographers is that they approach photography as if they are painting. Carefully thinking about the composition and techniques. Make sure everything is perfect according to a vision.
Japanese photographers are more spontaneous. Pictures look more like snap shots of everyday life. Often imperfect. Out of focus, crooked and blurry. They often just use a point and shoot on automode.
I just want to say that there are different ways to think about photography. I find the Japanese approach very interesting, because it uses unique properties of photography that is different from painting. Unlike painting you can take as many pictures as you want. You can take a camera everywhere. There are unique moments in everyday life that are beyond your imagination.
> There is a reason why companies like Google are doubling down in Silicon Valley real estate. They believe that the best, most innovative type of work are the results of organic discussions and unscheduled, unplanned, casual human interactions. So many cool ideas and new projects were the results of a casual lunch discussion even just random hallway chats
When was the last time you've used a big corp product that sounded like a water-cooler idea? Serendipitous innovation myth doesn't apply for these huge, hard-to-steer, profit-to-assets maximizing, financialized mega corps anymore. Nor creating 10 different messaging apps or torturing decade old products with growth hacking is innovation, it is cancerous growth born out of promotion-hacking. Water cooler schmoozing helps more with building the clout for that promotion than enhancing the productivity of the world.
They will make you work from office because they can, because that is easier on the managers and execs. They are doubling down on real estate because they have sunken costs, execs themselves probably have serious personal real estate positions and companies have a sizable chunk of h1b-captive employees that has to stick around anyway.
If anything, WFH had a positive impact on cutting through the bullshit. When speaking has more friction, the actual work has to do the talking, and that is a positive thing. You don't have to deeply enact a state of belief in other people's self-importance, as you get to say "oh fuck off" out loud when the meeting is off.
Going to the office is like going to the church of whatever objective function a company holds dear. You're surrounded by it for hours, and can't blaspheme around true believers and kool aid drinkers around that water cooler. At home, you're homed in a space that reminds who you are, your ultimate sovereignty and all the objective functions of your life, work being but one.
Around 2 years ago now, I took the plunge and bought myself an Ergodox EZ split island keyboard. Quite franky, it is the biggest quantum leap in the ergonomic experience of interacting with a computer I have seen since learning Vim. It is comfortable, effortless and fast. If you spend any significant time interacting with computers it is a complete no brainer to invest in optimising the IO channel between your brain and the machine.
Here is a link to my keyboard layout which you can freely use. It is optimised for Vim and Ubuntu use.
For best possible audio/video quality, join DIME torrents, which is where SugarMegs gets its stuff. http://www.dimeadozen.org/
Warning: If you're new to SugarMegs, prepare to lose a few days of your life. They have so much great stuff. Also, if you come across a set list that you can improve significantly (like you know all the song names), send them an updated HTML set list file, they really appreciate it.
It seems like in your case the easiest path might be to learn how the language works and see it for yourself. Otherwise, I think these posts are roughly in the genre you want:
Not to mention that the quality of life outside big cities is far healthier and more human scale and human paced. People are more focused on things like children than on Kubernetes — the world needs progress but boy does it take a toll. In the Bay Area I worked with people making $400K who said they decided they could only afford two kids. Meanwhile where I live now it’s very feasible for a schoolteacher couple to afford five kids, and not in privation. I like that Elon Musk exists but holy hell I do not want his life.
