I see this everyday. The problem is code is the wrong abstraction for the work we do. LLMs have solved coding, but they haven't solved systems, collaboration or system maintenance.

Edit: Since I seem to have touched a nerve - I've been working on a project to solve this: https://www.archme.io if you want to know my thoughts on the right abstraction

When you say "solved coding," what does this mean, what does it look like?

I have strong disagreement because it sounds like, by analogy or proxy, we have also "solved writing"

I would say it means you are building software but no longer are concerned with writing or editing literal code. Your concerns have moved up the stack to managing requirements, context, and verification processes.

Just to make this clear: if you can define a really good PRD and sophisticated technical specs, and a strong set of tests cases to pass, at the right level of architectural granularity, plus adversarial code review processes that triangulate and weed out most mistakes, SOTA agents can write the code autonomously, at or above the quality level of most human coding teams. I call that "solved" but only if you meet those context requirements. Which is still hard, not solved, at that layer.

Solving writing is not a good analogy IMO. Writing is for human consumption, and cannot be wrapped in objective requirements and verification processes. Certain forms of writing perhaps could be (can't think of one at the moment but I don't doubt some exist), and those forms might be good analogies for being "solvable" or "solved."

I agree with you, but it's a moot point for software engineering.

> Your concerns have moved up the stack to managing requirements, context, and verification processes.

This has always been the concern. "Add oauth to this app, there are no requirements beyond oauth working" has been an intern level task for ages. What makes software engineering hard is when the requirements start adding "well it has to use this oauth backend that isn't technically spec compliant, and the user is going to send some kind of random token you need to translate to oauth, and...". The problem isn't in writing code that will do the thing, it's figuring out exactly how that backend isn't oauth compliant and what chain of API calls I have to make to convert their random token into an oauth one, and etc.

Producing software that complies with a test suite isn't really novel. You've been able to outsource that forever. This falls apart in the same places outsourcing does; I'm sure India/Phillipines/etc/ is more than capable of iterating on code until it passes a test suite.

> no longer are concerned with writing or editing literal code

I'm not typing keys, but I am very much still concerned about the quality and nature of the code. Coding to me is more than pushing keys

> if you can define a really good PRD and sophisticated technical specs

I still believe we cannot waterfall software, the idea seems like taking a step backwards. How often do we learn about an unforeseen complexity only after getting into the implementation?

In my experience with agents, it's better to be iterative and in-the-loop. Start with a decent description, have them research the code/issue, write up an initial plan/design, work iteratively on writing code and updating design doc, review and finalize the code and markdown. Then future agents will have some resources to shortcut understanding the code base.

I'm not talking about not just punching the keyboard to type out the identifiers in the code. I'm talking about not making decisions about most classes and functions, most type definitions, most of the weedy logic within most modules, etc. If it can't be described in natural language as requirements, or in a typescript type or other data shape DSL (I think key data model types are probably still important to own), it's probably too weedy.

Natural language test cases still define the expectations both at the product and architectural level and are essential for triangulating the agents on successful outcomes.

A requirements and verification approach with agents is not waterfall any more than TDD is waterfall. Does thinking ahead and doing some planning equal waterfall? Does describing how a feature works to an end user, and making some key technical decisions, before you build it, mean waterfall? Does having some sense of what you're building first mean waterfall? With agents, you can specify (with PRD and technical specs) what you THINK it should do, and in minutes or hours or at most days, have the result, which you then learn from and iterate. If you didn't fully think it through, the agents will do one of three things: 1) decide for you, which you learn from 2) stop and ask, which you learn from 3) introduce bugs, which you learn from.

It's extremely iterative.

How are you thinking about technical debt in this new agentic age? (also markdown debt?)

It's one of my bigger concerns and I use this iterative approach to try and reduce it... "go look at the ./cli directory and generate me a report of inconsistencies blah blah..." or "review ./research/something.md, validate consistency, correctness, claims, ..."

... pseudo prose, have we made that a thing yet like pseudo code?

"solved coding" is type of thing you say if you want to sound smart.

Saying that LLMs have "reduced the cost of coding" would be boring. And using your analogy, pencils, typewriters and computers have all reduced the cost of writing, but writers are still around.

I'd separate "coding" from software engineering. Software engineering is not solved. The new challenge in software engineering is how to properly use AI to maximum leverage and nobody has solved that.

But, "coding" is, except that solving it means using techniques that are still barely understood and barely described today (something I'm hoping to be able to communicate better myself about). It is now possible (at least for most consumer software, I would not say applications where human life or extreme risk is involved) to work exclusively in the domain of natural language requirements, natural language architectural/design decisions, and natural language test cases, and deliver product of equal or better software quality than average human code authors could have produced. Even in a language that you plausibly don't actually know how to code in, because the language itself can often be separable from the requirements and verification process.

If someone who doesn't know a language or framework can plausibly produce better software using it, and faster, than a team of people who do know the language/framework (and I would strongly argue that is more than plausible now with models like Astra and Fable) then it's not just "coding is less expensive."

It's like saying computers make complex calculations less expensive. That's true, but it's missing the real paradigm shift.

I'm not trying to sound smart, I just think it encompasses what most people now acknowledge. LLMs write code faster and generally better than needed for most cases, and that seems to be becoming the prevalent opinion seeing as most folks have stopped doing PR reviews.

The problem is, without PR reviews & strict oversight, we're losing knowledge, system design & control of our codebases & products. Which is why, IMO, the coding is solved but the other parts which used to be so tightly coupled to programming are cropping up as their own issues.

This ties into my experience: greenfield projects turn bad way faster than legacy ones, because AI has no established patterns to key off and it does increasingly dumb shit.

"Reducing the cost of coding" would imply that you still have to code, but with current LLMs you no longer have to. You can write 100s of thousands of lines of code without ever having to touch a single line of code. Literally "here is a git repo, here is an issue, please fix".

You might still need to nudge the LLM in the right direction or stop it from going off weird tangents, but none of that involves touching actual code yourself.

What is coding? Pushing keys or knowing the right keys to push?

Ai can push a lot of keys very fast, but not always the right ones

if they need to be reminded to follow the coding standards, visible in the very code they are working on, what has been solved?

> What is coding?

Opening the IDE and typing program code. With LLMs you don't have to use an IDE, you don't have look at program code, you don't have to care about coding standards. You ask the chatbot to write you a program/feature/fix and chatbot does it.

Is chatting with a chatbot still "coding"?

The part that isn't fully solved is just the software architecture side of things, do you want library A or library B, or write it all from scratch? LLM can do all three, but if you aren't careful it might go down a route that you don't like. But that again can be fixed with chat, "replace A with B", not coding.

So lets say you plan the software architecture and take a person off the street, give them a frontier model and tell them to clone Figma given your architecture

How far would they get?

As a developer, a good way to reframe this and step away from your own domain expertise, is to read Terence Tao's conversation with ChatGPT and ask yourself "could I have done that?"

It solved coding as in it's significantly less of a bottleneck than it was before.

At least that's how I experience it. In the before times each non-trivial code change had a real opportunity cost as it would easily consume two days until I could even estimate whether this is worth looking deeper into.

Given a working system, like a web service or services, and their code, non technical people can now make effective changes with LLMs.

For example, nobody on our team writes manual code anymore, we have basically set up a harness where an engineer types up the requirements for a change, the system implements it given certain constraints, we have automated unit and integration tests that are ran, and if any errors pop up, they get fed back into the loop until fixed.

But to do that, you need to actually know what you are doing - you have to have good instructions to keep the agents in check and not start making mods outside of their bounds especially when the issue is with a dependant service that is causing errors.

I'm well on my way down this path too, but that is a process implemented in markdown. I don't see coding as solved, or a solvable thing at all, like writing

To solve something, there must be a defined problem, what is the problem that was solved. Or perhaps it is just "coding is solved" is the turn of phrase de jour be ause we haven't yet found a more succinct and accurate way to describe the paradigm shift

When it comes to non technical people using Ai to build things on code, the outcomes are on average pretty poor, which i see as evidence that the driver and their expertise behind the Ai matters a lot. A notable example is Terence Tao's conversation with ChatGPT, us math normies could never have done that. The same applies to coding agents ime

Writing is a means of expression and communication. Code can be those things, but that's not its primary purpose.

I think "solved coding" is taking it too far, but for many projects, the mechanical aspect of it has been removed or reduced greatly.

LLMs will have a much harder time "solving writing", because they cannot develop their own style and so are severely limited, creatively. This is less important for coding.

I agree that writing is much harder, largely because there is no way to get concrete feedback for "does it work"

I still think they produce shoddy or sus code too often, an artifact of the current generation's training to try anything and everything until it "completes the task". They have a hard time even with that concept, which is part of "coding" imo

maybe it’s more like “abstracted” away, in the same way higher-level languages “solved” needing to code via machine instruction sets. Higher level languages allowed humans to think more like themselves. AI puts another abstraction in front of the outcome, making it even more generally open to human thinking and less defined by the need to give machines exactly what they expect.

Today, human-language outlines / briefs / prompts are “compiled” to code which is itself then adapted to hardware. We are stretching less and less across the divide, doing less and less work on the terms of the machine. Now the farthest we’ll stretch is often formatted markdown - the most basic application of machine-parseable structure to very organic human thinking. Because we’re given the chance to be less precise, coherence suffers.

even setting aside the artistic side of the profession, I see these things make all sorts of basic mistakes, so even the mechanical part doesn't seem "solved" ime

Outsourcing solved coding a long time ago. You can go on Fiverr and prompt a real human developer for $2/hr - even cheaper than LLMs!

Wow, I think you have never had a chance to communicate with someone selling their expertise for $2/h. Unless your time is also worth $2/h, I think you would get better results prompting local 27B model.

Same problem though :). If an overseas employee knows the code and you don't, you have ceded ownership, problem solving ability & future direction of the project.

>The problem is code is the wrong abstraction for the work we do

My team recently spent two weeks on a wild goose chase trying to figure out why TensorFlow Lite was generating nonsensical OpenCL kernels. Well it turns out that LLVM had a few bugs in the RISC-V assembly for our platform that was leading to silent garbage. It took combing through assembly dumps, hexdumps, a lot of pain staking debugging, and going through the TensorFlow Lite source code to to track this down.

In your opinion, if code is the wrong abstraction to be working at, how do you approach this scenario?

Fair question - IMO it's the wrong abstraction for building and collaborating on a new product with a team.

To your point, it's not the wrong abstraction for solving code level bugs. Just like python is not the right abstraction for solving memory corruption or pointer mis-alignments.

Even if they "solved" that, the problem is it's the LLM that "knows" it, not the team.

Which is really the same problem with coding.

The agentic model of it just taking over and doing everything is poisonous to effective long term team work.

We're well past the point where it's about the quality of the work they produce. It's the way they integrate (or rather, don't) into human practices.

See my edit on the right abstraction - how do we get humans able to collaborate with agents effectively, where knowledge flows both ways, without needing a human to read 10k LOC, or even just lots of long LLM responses, to follow along

I have been trying to define "understanding". Is it when you can predict something that you understood it? Or maybe when you can explain it? Or how about when you can control it? Or invent it.

This time I add another definition "when you can own it".

When you can own it - I like that. Humans are still needed to own systems and drive/direct changes. The question is how can we collaborate as organizations and teams to still own it when we don't write the code and can't keep up with the output

I find it helps to imagine what is/isn't solved (however, we choose to label it) by an indefinite number of cheap junior-developers.

Except a little worse, since they were raised alone in a library, act mostly the same, and have harsh limits on personal growth.

tbh, I don't think of LLMs as junior-devs. Maybe 6 months ago, but now they are v. senior code-monkeys. They are masters of their craft, but their craft is narrow and lacks big picture, collaboration, org goals, etc.

Thanks my friend - I'm not an AI evangelist but I think it's pretty clear that AI codes very well at this point. So well that I'd hazard a guess that most software engineers don't write code anymore. Sure, maybe a professional could do better if they spent way more time on it, but it has always been a tradeoff we programmers have had to make. Getting things done + technical debt or writing the perfect code and over optimizing. And LLMs write working code, so the rest is just semantics on that scale.

Code being the wrong abstraction is similar to assembly being the wrong abstraction if you're trying to write a web browser game. Sure it's possible to do it and yes, you may end up with more optimized code, but it's not necessary and much slower to do that. Just use Javascript.

And now, people are building larger applications & functions quicker, and it no longer makes sense time-wise to look at Javascript for-loops when understanding the work. And that's because those for-loops are generally correct, if maybe a bit under optimized. Instead, engineers need a new layer of abstraction, to understand what the code is doing without having to read each line.

LLMs are faster until the technical debt comes due, at which point all supposed productivity gains evaporate and you're now 10x slower than doing anything by hand. They have a use if the debt never comes due, eg. for one-off scripts or for prototypes you will responsibly dispose of. The debt does come due in production, very quickly.

LLMs write code that compiles, which they accomplish mostly by robotically attempting the task and repeatedly fixing compile errors in a loop. That's a very narrow definition of "code that works", and I would argue it is the starting point, not the finish line. My definition of code that works is more like: secure, stable, maintainable, efficient, performant, effectively bug-free, with an ergonomic interface. LLMs fail on every single fucking count. I routinely observe generated code that leaves trivial 10x or 100x gains on the table, while having severe deficiencies in every measurable approach.

Given that you talk about assembly as a bogeyman, though, I gather that you're from the generation of "software engineer" who was already writing insecure, unstable, unmaintainable, 100x inefficient, 100x non-performant, bug-ridden JS for everything. LLMs can replace this class of people, it's true.

I couldn’t agree more. Code review is integral to engineering, to sharing system understanding, to building sustainable systems.

Something is missing in the new ai bot review paradigm we’ve all sleepwalked into.

I’ve been building Archme.io for this reason. PR reviews for the age of AI

I couldn't agree more. I've been building something that is trying to solve this problem, an abstraction layer for code that you, your agents and co-workers can all use together. I would love the chance to talk to you and get feedback if you're willing.

My email is in my bio

RankPic (https://www.rankpic.info) is an app to help users crowdsource their best photo. I've been building over the past 3 years & it's grown into a lovely community of people who help each other pick their best pictures for dating apps, professional photos etc.

I've seen some pretty fun novel use cases, such as (multiple!) people using it to pick out glasses, wedding invites & so on.

I recently completed a leaderboard function that cross compares photos from different tests using Claude, which was really impressive and scared me for my day job..

Hey Ben, this looks like a neat project/app! I'm always curious with apps like this about how financially viable it is. Seems to be a fair solution to be able to generate credits yourself by ranking and then buying more credits to get ranked, do you have any numbers or insights you are willing to share? (or have shared somewhere else before?)

Hi, thanks!

That's a great question. It is not making anybody rich, but I cover my costs and make a few hundred extra a month. I think on the order of $500+ MRR.

Besides buying credits there is also a pro membership where people choose which demographics rank them (age, location, gender), can have more tests, larger test sizes and always be at the front of the queue for ranking.

This is cool. It could be generalized into arbitrary ranking (not just photos).

Questions: how do you plan to deal with people that will randomly rank other's pictures to get credits quickly (or even build a script to do so)?

Hi! Sorry for the late response, went to bed and forgot about this post. Thanks :)

We check to make certain people aren't ranking formulaically, and if they are we invalidate their votes and ban them automatically. We also check how close people's rankings are to the consensus retroactively and do some fancy anti cheating patterns there as well.

Very interesting! Have you also thought about using an invitation system (like the one on lobste.rs - https://lobste.rs/about#invitations)?

Would you mind to tell a bit more about the exact logic and algorithms you use for those checks? I'm working on project in a totally different space (offers) but it shares the same idea where users vote on other users submissions and get credits for doing so (to incentivize participation and keep it fair). I haven't really worked out how to prevent abuse, so I'd be curious to learn from others. If you can share things (even just resources) I'd be super happy. Feel free to shoot me a mail to [email protected] in case you don't want to make things public.

RankPic (https://www.rankpic.info) is an app to help users crowdsource their best photo. I've been building over the past 3 years. It's grown into a lovely community of people who help each other pick their best pictures for dating apps, professional photos etc.

I've seen some pretty fun novel use cases, such as (multiple!) people using it to pick out glasses, wedding invites & so on -- https://apps.apple.com/us/app/rankpic-photo-ranking/id160299... (ios) -- https://play.google.com/store/apps/details?id=app.rankpic.ra... (android)

Great idea, validate before getting validation on social media. nice.

Thank you!

I founded a startup based on this idea. Track purchases with credit cards and sum things up on a monthly basis. Unfortunately couldn't find a grocer to take me up on it mid pandemic, but I want to try it again in a few years if no one has made it work yet.

Thanks for asking! The community is about 15,000 users, maybe 100-300 DAU, tests complete in as little as 10 hours for 20 ranks or up to a couple days if you have a lot of niche filters set for your rankers (age, sex, location).

The best ROI I have found is on Apple and Android app stores. I tried Facebook, Tiktok & others and have found them very expensive with little return (though I admit I'm not marketing expert).

As for Photofeeler.. I actually created this app because I found the way it rated me 1-10 fairly harmful to my mental health. RankPic has users rank your photos best to worst, so it's just against yourself. Additionally you can get more mileage out of each test as you can upload up to 6 photos at a time.

Interesting, that is a good size userbase! And good argument regarding the ranking vs rating - did not consider this.

I have a site, app.rapidata.io/compare which has a related feature, allowing you to have 100 people vote between two images based on a question (it was made as kind of a demo for our underlying product, but maybe you find it interesting). These are mostly random people though and it does not as detailed filtering as you have.

RankPic (https://www.rankpic.info) is an app to help users crowdsource their best photo.

I've been building over the past 3 years & just recently monetized and crossed the $500/m mark through a Pro subscription. It's grown into a lovely community of people who help each other pick their best pictures for dating apps, professional photos etc.

I've seen some pretty fun novel use cases, such as (multiple!) people using it to pick out glasses, wedding invites & so on

-- https://apps.apple.com/us/app/rankpic-photo-ranking/id160299... (ios)

-- https://play.google.com/store/apps/details?id=app.rankpic.ra... (android)

How does an app like this work in the beginning when you have no/few users? Were you ranking images yourself or hire people to it?

I got all my friends to go on it and spent lots of time ranking people myself.

I also started things off with "fake" tests so that people had other people to rank at the beginning, otherwise folks would just see a dead app and drop off.

It was a really big moment 9 months in when someone's test completed entirely without me ranking. It was a lot of work & I'm really glad to be at a point now where I get to watch people rank each other without me.

How did you design and build your homepage? I find that building the landing page and making it look like a professional, beautiful design is one of my biggest hurdles. I'm an experienced web developer but without a design to work off of - and especially accounting for mobile and dynamic sizing - I really struggle with this part of the work so I'm wondering what other people's workflows are for it.

I used fiverr to make assets, and then either use fiverr to make a design that I can easily build with one of the website building tools (wix, etc.) or I try to build it myself.

The easiest way is to find a landing page you think looks good and use it as inspiration with your own colors, assets, etc.

In case you're looking for a designer, I can help you with the design especially mobile as I have worked on mobile apps that are live on app store and play store. feel free to check out my work : https://monadile.framer.website/ You can also reach out via email : monadile.design(at)gmail.com

Thank you, but I probably will do this myself. For what it's worth - and I'm saying this constructively, not to offend - your website is illegible due to contrast issues. White titles and light grey text against a light blue background. I was sure it must be some kind of error but I turned off all my extensions and loaded it incognito in chrome and nothing changed, so, I don't know what conclusion to draw there. It just doesn't make a very good impression for a designer.

Out of curiosity, why did you start this?

I've always used photofeeler.

Thanks for asking! I also used photofeeler and found the way it rated me 1-10 fairly harmful to my mental health. RankPic has users rank your photos best to worst, so it's just against yourself.

Additionally it allows users to get more mileage out of each test, when you can do 2-6 photos instead of just one at a time.

Anecdotally I've also heard it is a more fun experience for the rankers.

Very smart the best to worst ranking. Well done!

Thank you! The ranking queue was a fun thing to implement.

That answer is very inspiring, thanks :)

Are you up for a digital coffee? If so, my email is in my profile.

While I haven't been an entrepreneur, I did some side projects. Would be fun to chat about it.

Thanks! Emailed :)

I think "raters" should be able to skip because some categories are strange.

Like I just got "Category: Monster girls" with 2 pictures of anime.

I also got "Category: Dating" with 4 family pictures of kids (I reported this one)

Without any offense to you, the creator of this app. It's obvious a lot of care went into it and you wanted to create a better product than what is out there. Even considering the mental impact such ranking could have.

However, I genuinely feel that the need for this app is what's wrong with society.

What's wrong with this? Seems like a more independent alternative to asking a friend which picture looks better.

I personally would prefer a randomly selected load of strangers to vote on, say, my best corporate headshot to display to the general public rather than my friends who are a small and biased group.

I guess the people that self-select to go on a photo ranking site may not be representative of the general public, though are probably better than the (all male) 10ish engineers and 3 accountants that I socialize with on a regular basis.

No offense taken, I also have many qualms about the role looks and photos play in our society. The destruction & gamification of social interaction by tech is a genuine harm.

Unfortunately it's what we're dealing with right now, and people need feedback to be able to play the game.

RankPic - www.rankpic.info (available on Android and iOS, website is.. meh)

App that lets you crowdsource people's opinions on your photos. Unlike other websites out there, it doesn't rate you 1-10 but instead ranks your photos against each other so you don't have to feel bad about yourself :). Very useful for dating apps and other social media, esp. if you don't have easy access to friends' opinions.

I've had lots of friends and users tell me it's super useful, but I'm in a slump on building it right now.

worthy cultural events.. what? You are missing out on like all of NYC