Calling open-weights as open-source in marketing materials is the usual misrepresentation. But now with the restriction on commercial use (which is against opensource definition) it is not even open-weights, technically it would be more accurate to call it weights-available.
The problem is there is no good metric to describe code quality, it cannot be RLd and that's likely why latest models have such problems with slop. It is great to raise the conversation so that labs focus on this more.
Picking specific metrics will probably not work, it would be a mix of Goodhart's law with Bitter lesson. Maybe picking and labeling quality repos, having whole suite of metrics as input features and training some traditional AI classifiers to steer the LLM training.
He used LOC, and it isn't bad. Just this week I had a LLM do a small task, and it produced 500 LOC, every little detail beautifully abstracted out. But 500 LOC for such a simple change looked suspicious to me. As everything must pass human review, I re-wrote it to see what happened. The result was 100 LOC. No human wants to review 5 times a much code.
The issue really isn't "there is no good metric" - when I saw 100 LOC vs 500 LOC for the same thing there was no argument. The problem is Goodhart's Law. Whenever we use a metric use like LOC as a reward function for humans, the result is a disaster. I have no doubt that's true for LLM's too.
GitHub is really two separate products: a code forge for teams and a social coding platform to collaborate on open source.
When I use the social platform, it is mostly as a read-only website. I am occasionally commenting, opening issues, PRs etc. The slowness and annoyances do not matter that much.
But I live in the team tool, where the annoyances and paper cuts are painful. Besides poor craftsmanship, there is often tension between the simpler design for social which sacrifices usability for pro use cases.
The social platform, which has huge network effects and is really hard to replace, is basically a loss leader for marketing the team product. I wonder if it would be viable for another company to create a team tool that stands on its own by outclassing Github like Linear outclasses Jira? That should be easier to replace. And Microsoft could keep running the social platform for the time being, where the adoption of alternatives is much slower and requires a much longer time frame.
This is quite puzzling. Back in the day I saw the breakthrough with Opus 4.5, and since then was very lightweight on CLAUDE.md, trying to declare intent, rather than prescribe imperative instructions. If it was off the mark, I would see clear mismatch, which gave me opportunity to correct, usually to clear and strip context.
Starting with Fable 5, if it goes off the rails, it is more difficult to correct it, because it is overall wrong, but covers its tracks with plausible sounding arguments, so it is hard to pin point and correct.
Now, this article points out techniques that were useful to rely on models more, but those peaked at 4.6. Now according to benchmarks Opus 5 is on the frontier. But when when it has looser reigns, it ends up gaslighting me even more with abstract word soup than any model before.
As a counter argument for the compiler perspective: what Cursor did is basically transpiling a project to another language, which is a form of compilation.
If you distill the feedback into a spec (probably a combination of both docs and tests) which comprehensively covers the functional requirements, then you could treat the code as cached materialization. The value of generating code again is targeting different non-functional requirements, like ability to run in different environments or catering to different performance characteristics.
Specs might become one solution for coping with the need to review increased volume of code. A spec is a higher level of abstraction than code, which is a higher level of abstraction than machine code. The industry made the transition to higher-level once, paradigm is changing so it might happen again.
The workflow I imagine is either deriving specs from the conversation or reverse engineering the code to spec, review and edit the spec which should be tighter and much more compressed, then deterministically compile to code. Of course we don't want to be spec-first only, that would be going back waterfall, but doing iterations back and forth.
Now, Claude is not a compiler because it is closed and non-deterministic (they do opaque processing on server, hiding reasoning tokens), but LLMs might be. We refer to a piece of code from npm/pip by name to get some code by downloading it. We then have lockfiles with hashes to ensure integrity. Currently we are vibing it, but in the future we could refer to a piece of code by prompt/spec and getting the code by inferring it. To ensure integrity, the lockfile would be hashes of open weights and inference code (and ironing out implementation details like non-determinism due to GPU scheduling, etc.).
It's funny how you describe something very close to what I was attempting to design. It was meant to trickle down from tickets to deterministic code. Data wise, it should be like a pyramid of tickets diluting layer by layer into leaf nodes which were "implementable" as statements in code. I'm not sure if that description makes sense read by someone else.
I think what made sense was envisioning a nanoswarm of LLMs (anticipating ASIC performance) diluting specs into semantic logic nodes, but a conversion from these to deterministic code made the vibe coded experiment come to halt. Your lock approach could be a shortcut to that.
Totally off-topic: it's quite intriguing that you can feel once friction starts building during a design phase. Suddenly everything slows down. I wonder if it's quantifiable and therefore can identify "wrong" design choices made by either humans or LLMs.
(Hoping some claw bot pics this up to finish my idea on my github tix repo in the initial-design branch, as main is empty ~ MPL2)
This resonates, I can imagine the development process as a cascade: we get business goals and user feedback, product is defined which can be pretty fuzzy, designers define UI, engineering works data models and tech constraints. Inputs at the top can be fuzzy, but each node in the graph applies some intelligence and clarifies it.
Related to the friction you mention, since the flow graph of the context is not preserved when it gets transformed into code, I find LLMs getting confused and making worse changes, because the explicit decisions with good reasons are getting mixed with implicit decisions from LLM surfaced during designing/planning. Sometimes when things change the code should change radically to match the new reality, but LLMs often "play it safe" by accumulating code and adding to tech debt. With a flow graph when change is made in the upper levels, it could propagate and allow decisions in lower levels to be revisited.
> A spec is a higher level of abstraction than code, which is a higher level of abstraction than machine code
No it’s not.
Just like a quick doodle is not an higher level representation of the Mona Lisa. Sure for someones that knows the latter, it can suggest it. Or for someone that doesn’t know it, it may provide some basis of conversation. But it’s not the real thing. You can’t provide a doodle of something and expect an artist that hasn’t seen it to paint it.
Doodling’s value is that it lets you iterate on ideas without the accidental constraints that comes with the implementation (choosing paints, finding references materials, deciding colors,…). Some necessary choices are kept for later, while you decide on the most important ones.
So a spec is useful in designing software, but it’s severely lacking in implementing it. And the decisions made in implementation are as important as the ones made in the design. Even more after a while in production.
This might work if we have completely new harnesses for this, because the current one’s intentionally introduce non-reproducible context and tools.
The current way we’re doing SDD is extremely pointless. You reach a decision point that is under specified, make a decision, update the spec and continue the implementation. There’s no guarantee that the implementation followed from the updated spec, and not from the decision.
I agree, another issue I have with existing SDD frameworks I've seen like spec-kit or openspec, is they seem to miss the mark by coupling specification (what the software should be) and execution (how to get there). If the description of execution is needed, it means the spec is weak. As a rule of thumb if a framework defines execution concepts like tasks or tickets, that is a red flag for me.
prompts are not specs because most often, prompts are underspecified. You can prompt fizzbuzz well enough for it to be functional, but not ffmpeg. So the distinction is less about Claude's capability, more about the quality of the input. Garbage in garbage out
I think something can be considered a spec to some degree if outcome was derived from it.
It might underspecify, but that is just a quality of the spec. And as long as there is any discretion at implementation time, then the spec necessarily underspecifies to some degree. The goal of a spec is often to be clear on key directional concerns. Consider RFCs.
There's a trade-off between specification and implementation discretion. But if you think about starting at one end ("an internet forum") and then incrementally expanding the specifications ("it has topics with many posts in a linear paginated list"), there's some point where everything you care about is in the spec, and the rest is left to implementor discretion.
That's what I'd consider a good spec and a good LLM prompt.
and LLMs are really good at filling in the blanks. It's like a really big import on generic application knowledge, and if you draw the circles well enough, it will draw the rest of the owl.
You can reference other specs in a prompt. Video codecs are well specified with hundreds of pages of specs, they are probably in some form in the weights already, but you could also reference them and hash them as inputs directly to create a functional alternative to ffmpeg.
i meant to make the distinction between 'prompt' and 'spec' in that prompts are something that a human types in by hand. Interesting programs are a combination of canon and novel input
Gotcha, yeah if a prompt does not produce the result you want, you need to add more details to it, depending on the scope it can be a lot. When it includes all sufficient details it becomes a spec.
Cool project, I like the idea of having tmux-compatible CLI. I used Zellij to get better UX, but many agent tools integrate with tmux. This way agent tools can still integrate tmux as a defacto standard for programmatic interface while having better interface for users.
But I wonder if tmux/rmux design is suboptimal since it couples session persistence and window management together. Do you have an opinion the design which separates the responsibilities? An example that pioneered this approach is abduco, and libghostty-based zmx being a modern implementation.
I thought I was too zealous because I created 11 crates respecting single responsibility principles.. I would have been all in to risk it further and separating responsibilities for session and window management but it would have diverged too much from tmux. I wanted a 1-1 port for now
correct... it's sad but recognition is a lot. After putting months of efforts, it's a risk I didn't want to take. And following tmux was like a red thread of life to code. Maybe in the future !
Reviewing large volume of code is a problem. In the pre-LLM era, as a workaround to occasionally review large PRs, I used to checkout the PR, reset commits, and stage code as I would review. In the first pass I would stage the trivial changes, leaving the "meat" of the PR that would need deeper thinking for later passes.
With the increased volume of code with agentic coding, what was once occasional is now a daily occurrence. I would like to see new kinds of code review to deal with larger volume of code. The current Github review interface does not scale well. And I am not sure Microsoft has organizational capacity to come up creative UI/UX solutions to this problem.
GitHub seems entirely uninterested in improving the code review experience (except maybe the stacked PRs thing if that ends up shipping) for well over a decade now.
Things that I’d consider table stakes that Phabricator had in 2016 - code movement/copying gutter indicators and code coverage gutters - are still missing, and their UI (even the brand new review UI that also renders suggestion comment diffs incorrectly) still hides the most important large file changes by default.
And the gutter “moved” indicators would be more useful than ever, as I used to be able to trust that a hand-written PR that moves a bunch of code around generally didn’t change it, but LLM refactors will sometimes “rewrite from memory“ instead of actually moving, changing the implementation or comments along the way.
