I was there in the 90s. I built a few bad static HTML pages. It wasn’t hard. There are lots of stories of non-CS / non-technical people making stuff from the dotcom era.
Making a dynamic page was harder. Integrating with a payment system was almost magical; there’s a reason PayPal became big.
But what was truly hard, and continues to be hard, is building a page, either static or dynamic, that people actually want to visit.
Building static pages that worked in both Netscape and IE 4, and could function well with constrained dial up speeds, may not have been "hard", but it did come with a number of challenges.
Give it another 10-20 years and your 2 hour podcasts will be 30 minutes of morning zoo DJ banter, 10 minutes of guests, and 1.5 hours of ads.
We’ll have reached peak 90s all over again. With any luck we’ll avoid recreating the conditions for another Nickelback and can stay in the weird zone where Trip Hop and pop punk could chart at the same time.
The 00's podcasts I listened to were often in 2-3 hour episodes, rarely well scripted (or scripted at all?), but a lot of fun and very amateurish. I re-listened to several entire series recently and the episode lengths were the only thing I think was worse than in newer podcasts.
On the other hand, if ads etc gets too annoying, I already have run all my downloaded podcasts through whisper to get transcripts with timestamps. Running some LLM to find ranges to delete would probably be quite easy. As a bonus I would be happy to also cut out all the filler repetitions that seem popular these days ("yes, X, I absolutely agree, [repeats everything X just said]"). Could probably cut 1 hour episodes to 20 minutes without losing any content.
“Autofishers” are large boats with nets that bring in fish in vast quantities that you then buy at a wholesale market, a supermarket a bit later, or they flash freeze and sell it to you over the next 6-9 months.
Yet there’s still a thriving industry selling fishing gear. Because people like to fish. And because you can rarely buy fish as fresh as what you catch yourself.
Again, it’s not a great analogy, but I dunno. I doubt AGI, if it does come, will end up working the way people think it will.
This is written with the idea that the exponential part keeps going forever.
It never does. The progress curve always looks sigmoidal.
- The beginning looks like a hockey stick, and people get excited. The assumption is that the growth party will never stop.
- You start to hit something that inherently limits the exponential growth and growth starts to be linear. It still kinda looks exponential and the people that want the party to keep growing will keep the hype up.
- Eventually you saturate something and the curve turns over. At this point it’s obvious to all but the most dedicated party-goers.
I don’t know where we are on the LLM curve, but I would guess we’re in the linear part. Which might keep going for a while. Or maybe it turns over this year. No one knows. But the party won’t go on forever; it never does.
I think Cal Newport’s piece [0] is far more realistic:
> But for now, I want to emphasize a broader point: I’m hoping 2026 will be the year we stop caring about what people believe AI might do, and instead start reacting to its real, present capabilities.
It clearly didn’t come out of game dev. Many people doing high performance work on either embedded or “big silicon” (amd64) in that era were fully aware of the importance of locality, branch prediction, etc
But game dev, in particular Mike Acton, did an amazing job of making it more broadly known. His CppCon talk from 2014 [0] is IMO one of the most digestible ways to start thinking about performance in high throughput systems.
In terms of heroes, I’d place Mike Acton, Fabian Giesen [1], and Bruce Dawson [2] at the top of the list. All solid performance-oriented people who’ve taken real time to explain how they think and how you can think that way as well.
I miss being able to listen in on gamedev Twitter circa 2013 before all hell broke loose.
I use a notebook extensively for certain kinds of work and problems. I’ll point out specific ways that I use it to answer your questions.
> The post here mentions hypotheses, but I don't do experiments for the most part.
Debugging any hard bug is essentially making a series of hypotheses and testing them. I use a notebook to keep track and make notes when I’m knee-deep in some hairy bug.
> It mentions writing down in the notebook before writing code, but I can't test my notes, I can't really send my notes for code review.
I use a notebook primarily for design work, especially algorithmic design work.
It’s really handy for numerical stuff where I often want to transform some expression or equation and prove to myself that it’s equivalent or has certain properties.
It’s also handy for working through any algorithm where you’re manipulating a tree or a graph.
> I guess you could use it for design, but you'd lose all the advantages of word processing such as editing, links, context, etc.
I find it much faster to sketch the things I mentioned (especially diagrams!) with pen and paper when working through them. If I need to present the work or share it, I might scan the notes or I can polish them in a word processor or slide deck or whatever.
For what it’s worth, my background is in a computational science (not CS) and I do quite a lot of work on numerical and algorithmic problems that come up in actual hardware and sensors. I also like to work on compiler-y things in my spare time.
Ultimately you end up using tools that are useful for you. So none of this may have any value for your work. But hopefully it answers what someone might write in a notebook.
I really tried to like the podcast. It’s been a few years, so maybe it’s improved.
The topics were good. The guests were great.
But Bryan Cantrill was just terrible at letting his guests actually talk.
Bryan, if you’re listening, please let your guests talk. We have a large amount of content on YouTube if we want to hear the Bryan Cantrill take on, well, anything and everything. And it’s often amusing and sometimes right.
People don’t tune in to a podcast with guests to hear the host pontificate. They tune in to hear the guest, and sometimes the guest/host dynamic. When the host talks over the guest, you don’t get either.
After the Jonathon Blow episode, I gave up. Dude had interesting things to say about C, C++, and Rust, but most of what we got was Bryan talking about Rust. I guarantee anyone tuning into the Oxide podcast knows Bryan Cantrill’s opinions about Rust. And firmware. And Oracle. And Linux. Etc. etc.
Well, a couple of things. First, the Jonathan Blow episode[0] was over six years ago. Second, it was nearly a three hour conversation -- I don't think I can be accused of not letting him talk? Third, I definitely remember that I felt I had to interrupt him to move the conversation along. Fourth, I had to pee really badly, I was absolutely freezing, and I was quite concerned about missing my flight to New Zealand that evening with my family for Christmas (which I damned near did) -- and I have no doubt that I was not at my best!
I do try to get better at this stuff, and I re-listen to our episodes to improve as an interviewer. If it's been "a few years", maybe you haven't listen too much to Oxide and Friends? I think we've had some wonderful guests and great conversations over the span of the podcast -- though I also have no doubt that it's imperfect, for which you have my profound apologies!
I appreciate the reply and that it's been a while; I will give your podcast another listen.
It wasn't just the Jonathon Blow episode; that was just the point where I said "this is frustrating." For what it's worth, frustration came from knowing that this could be really good: your perspective is valuable, your topics were interesting, and your guests were excellent.
I find this a common mistake that people with strong opinions have when doing interview/guest style podcasts or shows. There's really an art to it; it's not easy to engage guests, keep the show interesting, and let the talk move in interesting ways. That's why Terry Gross and Howard Stern, in very different ways, have had such long and storied careers.
But it's something that people definitely get better at, and I have no doubt that you have. Again, I'll give it another listen.
Is this only based on On the Metal? (If so, those are all from six years ago -- even the ones that were released a mere five years ago.) Please do check out Oxide and Friends[0] -- and feedback always welcome!
These sort of transparent answers are what make oxide and the people behind it such a fantastic company. Thank you for your wonderful contributions to the software and hardware community!
I disagree. I found the 'On the Metal' to be great balance between the guest and the hosts. And Oxide and Friends I would expect Oxide to be big part of it.
> There are plenty of papers out there that look at LLM productivity and every one of them seems to have glaring methodology limitations and/or reports on models that are 12+ months out of date.
This is a general problem with papers measuring productivity in any sense. It's often a hard thing to define what "productivity" means and to figure out how to measure it. But also in that any study with worthwhile results will:
1. Probably take some time (perhaps months or longer) to design, get funded, and get through an IRB.
2. Take months to conduct. You generally need to get enough people to say anything, and you may want to survey them over a few weeks or months.
3. Take months to analyze, write up, and get through peer review. That's kind of a best case; peer review can take years.
So I would view the studies as necessarily time-boxed snapshots due to the practical constraints of doing the work. And if LLM tools change every year, like they have, good studies will always lag and may always feel out of date.
It's totally valid to not find a lot of value in them. On the other hand, people all-in on AI have been touting dramatic productivity gains since ChatGPT first arrived. So it's reasonable to have some historical measurements to go with the historical hype.
At the very least, it gives our future agentic overlords something to talk about on their future AI-only social media.
Perhaps you're comfortable with a compiler that generates different code every time you run it on the same source with the same libraries (and versions) and the same OS.
I am not. To me that describes a debugging fiasco. I don't want "semantic closure," I want correctness and exact repeatability.
I wish these folks would tell me how you would do a reproducible build, or reproducible anything really, with LLMs. Even monkeying with temperature, different runs will still introduce subtle changes that would change the hash.
This reminds me of how you can create fair coins from biased ones and vice versa. You toss your coin repeatedly, and then get the singular "result" in some way by encoding/decoding the sequence. Different sequences might map to the same result, and so comparing results is not the same as comparing the sequences.
Meanwhile, you press the "shuffle" button, and code-gen creates different code. But this isn't necessarily the part that's supposed to be reproducible, and isn't how you actually go about comparing the output. Instead, maybe two different rounds of code-generation are "equal" if the test-suite passes for both. Not precisely the equivalence-class stuff parent is talking about, but it's simple way of thinking about it that might be helpful
There is nothing intrinsic to LLM prevents reproducibility. You can run them deterministically without adding noise, it would just be a lot slower to have a deterministic order of operations, which takes an already bad idea and makes it worse.
Please tell me how to do this with any of the inference providers or a tool like llama.cpp, and make it work across machines/GPUs. I think you could maybe get close to deterministic output, but you'll always risk having some level of randomness in the output.
It's just arithmetic, and computer arithmetic is deterministic.
On a practical level, existing implementations are nondeterministic because they don't take care to always perform mathematically commutative operations in the same order every time. Floating-point arithmetic is not commutative, so those variations change the output. It's absolutely possible to fix this and perform the operations in the same order every time, implementors just don't bother. It's not very useful, especially when almost everything runs with a non-zero temperature.
I think the whole nondeterminism thing is overblown anyway. Mathematical nondeterminism and practical nondeterminism aren't the same thing. With a compiler, it's not just that identical input produces identical output. It's also that semantically identical input produces semantically identical output. If I add an extra space somewhere whitespace isn't significant in the language I'm using, this should not change the output (aside from debug info that includes column numbers, anyway). My deterministic JSON decoder should not only decode the same values for two runs on identical JSON, a change in one value in the input should produce the same values in the output except for the one that changed.
LLMs inherently fail at this regardless of temperature or determinism.
Just because you can’t do it with your chosen tools it does not mean it cannot be done. I’ve already granted the premise that it is impractical. Unless there is a framework that already guarantees determinism you’ll have to roll your own, which honestly isn’t that hard to do. You won’t get competitive performance but that’s already being sacrificed for determinism so you wouldn’t get that anyway.
Agree. I'm not sure what circle of software hell the OP is advocating for. We need consistent outputs from our most basic building blocks. Not performance probability functions. Many softwares run congruently across multiple nodes. What a nightmare it would be if you had to balance that for identical hardware.
That is exactly how JIT compilers work, you cannot guarantee 100% machines code generation across runs, unless you can reproduce the whole universe that lead to the same heuristics and decision tree.
Once I create code with an LLM, the code is not going to magically change between runs because it was generated by an LLM unless it did an “#import chaos_monkey”
Making a dynamic page was harder. Integrating with a payment system was almost magical; there’s a reason PayPal became big.
But what was truly hard, and continues to be hard, is building a page, either static or dynamic, that people actually want to visit.
reply