I do a lot of interviews for my employer and anecdotally the current batch of college hires seems worse at answering the questions I give them than prior groups. I try to avoid leetcode style questions unless they tell me they've done competitive programming. Typically I ask a somewhat open ended question that requires implementing some complicated but not particularly tricky business logic. I used to be able to ask a few follow-up questions that added additional requirements but lately I've found candidates struggle to even finish the original question. It may not matter since the reality is LLMs could handle the sort of questions I ask just fine, but I do wonder what the long-term affects of this decrease in coding fluency will be.

I think being able to read a spec and implement business logic to that spec is a basic coding skill, just like being able to read and summarize an argument is a basic literacy skill. If people are worse at that it means they are less proficient at programming than they used to be. It might not matter since LLMs are very good now and will continue to improve, but it's strange to see the quality decline.

If The Great Game you mentioned is the one by Peter Hopkirk, it's fantastic.

If you enjoyed those books I can highly recommend John Biggins' Otto Prohaska series, which I've seen described as "a techno-thriller set on a WWI submarine". Biggins books are really fun and well researched.

Thank you! I will check them out. I've been liking the Dungeon Crawler Carl series that is discussed in this thread. It's light and entertaining. Also Children of Time series is pretty good.

I recently finished George Vaillant's book Adaptation to Life and immediately picked up the follow up The Wisdom of the Ego. Vaillant is a psychiatrist and was a director of the Grant Study, which followed about 250 Harvard undergrads from college until they died.

Both books are largely about how the psychological defense mechanisms a person uses play a large role in determining how successfully they adapt to and navigate life.

> LLMs are stochastic and probabilistic

So are humans. Every codebase I've worked on has duplicated code that has been written in slightly different (but hopefully equivalent) ways, often by the same person.

Most managers don't read the code of their direct reports. They just trust that the code is "good enough" and that there are enough other processes in place to catch bugs before they cause too much damage. I think I'm a pretty skilled programmer, or I'm at least good enough at interviewing to convince other people of that, but it's pretty obvious from comparing LLM generated code to code I'd write myself that in many domains LLMs are superior to me. They do make mistakes, but so do I, and those mistakes are eventually found and corrected.

We're in the very early stages of LLM driven programming, so it's hard to say how it will all shake out, but my anecdotal experience is that LLM written software is very reliable and easy to extend and develop. I have a side project that is about 90% LLM generated code (about 25k lines of production code and a similar amount of test code). This is a revenue generating product and I've had no issues with reliability, security, or performance.

For what it's worth this app is a rails app and I have no plans to switch to anything else. Rails works nicely, the LLMs extend it easily, and almost everything is I/O bound so I don't need C++/Rust level performance.

> I think I'm a pretty skilled programmer, or I'm at least good enough at interviewing to convince other people of that, but it's pretty obvious from comparing LLM generated code to code I'd write myself that in many domains LLMs are superior to me.

there's an important caveat here - LLMs absolutely can write better code than me, but they can also write far worse code, and often don't seem to be able to tell the difference. I find myself spending a lot of time prompting the bots to refine their code in specific ways that I only know about because I read the generated output.

    but my anecdotal experience is that LLM written 
    software is very reliable and easy to extend and 
    develop.

Extremely refreshing to read. The whole thing, not just this statement.

Too much discourse creates a false dichotomy of "awesome, wonderful, hand-crafted code" vs "shitty AI slop."

A lot of hand-crafted code is pretty bad. This is true even with talented engineers: there are often edge cases they did not think of... and in many cases, could not have thought of.

And AI code is pretty good if it's steered and vetted by a knowledgeable engineer. It is clear to me, and has been clear for a long time, that "talented engineer plus AI" is the winning combination and will remain that way for at least the near future.

*"talented engineer plus AI" is the winning combination and will remain that way for at least the near future.

Talented programmers will retire and Father Time takes care of the rest. Where will the new wave of talented programmers come from? I use AI, but I am a hobbyist.

I still review PRs, all of which are written by agents. I almost never have meaningful feedback to offer unless it's feedback on the architectural approach. And even then, a lot of architectural/coding patterns that I've been a stickler for in the past mean less to me now because those patterns served to create maintainable code for human beings, which just isn't a priority anymore.

At our company (big Rails/React monolith) all PRs do get a human review. Now, I use LLMs to accelerate my reviews, but there's still a lot of space for humans.

- LLMs struggle, or at least burn oodles of tokens, on big complex codebases. Managing tech debt helps the agents as well IMO. - If you ask an LLM to review a non-trivial pull request, it nearly always seems to find at least 10 issues. Many of which are not worth pursuing. Often, they would introduce a lot of extra complexity to harden the code against scenarios we don't care about or can never happen. So there's a lot of room for human judgement there. Typical scenario - the LLM reviewer finds 10 issues and I judge that 2-3 are worth pursuing - As you said, often the architectural approach sucks even with frontier LLMs. Usually from a lack of problem space / usage scenarios more than a lack of technical chops - LLMs tend to err on the side of overengineering the shit out of everything

I do not see how LLMs will be able to manage huge, hairy polygot codebases without humans-in-the-loop in the near future.

Given the pace of progress, I'd be a fool to bet against LLMs in the medium term future. But I think it's far from a given. LLMs themselves, and large codebases, are essentially many-to-many problems approaching something like O(n^n).

Most companies would rather have AI without talented engineer.

Maybe. The market will figure out if this is viable, I guess.

For smaller projects or short-lived projects, "AI without talented engineer" is a viable choice already.

But a lot of projects are big and long lived.

The difference in complexity between 1K lines of code and 10K lines of code isn't merely 10x. It's usually more like 100x.

So even if AI becomes 10x better than today (which seems reasonable) there's going to be a level of complexity beyond which you still need some kind of talented engineer-type(s) steering the AI.

Yeah let’s be real there was a ton of slop around before AI

I think there's aspects of this viewpoint that are true. I'm sure I'm less skilled at programming now in some ways than when I started using LLMs, but I'm a lot more productive and I can work on a broader variety of software than I could before. Writing probably weakened people's auditory memory, and calculators probably reduced people's mental math skills, but I don't think either are a net negative. It just requires some adjustment to get used to the new way things are. I also think my programming skills would come back pretty quick if I started programming without LLMs again, just as my leetcode skills come back pretty quick when I prepare for interviews.

I feel like LLMs have been nothing but a boon for me. I have learned so much programming from them. I have over a decade of experience, but honestly, I have always been a below-average programmer. Good enough for boring CRUD work, but not much more. Forget Leetcode, I'd probably get murdered on the easy questions.

So, if anything, I feel like my learning pace has been greatly accelerated. Your point about working on a broader variety of software is really true for me too. I am working on a game that I likely would have never even started had it not been for LLMs. Mind you, I am still doing the vast majority by hand. However, LLMs have been great for little tutorials, setting out a good game plan, etc. It's like having my own little private tutor.

I know enough to not let the LLMs do everything for me. If they do, then I know I will fall into the trap of something breaking, and then I lack the understanding to fix it. I think they should be used to make life easier, but not easy.

People are still interested in chess even though computers are much better than humans.

Not really unique to VC, similar things are happening to private equity with secondary funds.

I rarely use image search, but I went to look something up recently and I was shocked at how many obviously AI generated images showed up. I couldn't even find an image of the thing I was looking for and eventually gave up. Bing has the same problem.