It’s really not a concept you can express in idiomatic Python very easily. This comes from the actual generated assembly involving copies from global GPU memory into registers (slow, bandwidth saturates quickly) and back in between the cosines. If you can avoid the intermediate roundtrip that cuts the cost approximately in half.
One question I have here: I think this type of thing would be trivial to do in Rust with constructors, private fields, and newtypes. What am I getting on top of it?
For a single-language Rust project with a handful of invariants, not much. Rust's newtypes + private fields + Result-returning constructors are exactly the right primitives, and they're strictly stronger than Go's (no reflection escape hatch, no zero values to forge, exhaustive matching).
Where shengen might be more interesting:
1. Multi-language emit. If your invariants live in a Rust service AND a TypeScript frontend (or another backend), one Shen spec drives both and the build catches drift. Hand-rolled means writing the same smart constructor twice in two languages with no mechanism to keep them in lockstep.
2. Spec as the audit surface. The Shen rule for `tenant-access` is going to be much more expressive / concise than the rust implementation. The `tcb-audit` gate fails the build if generated code is hand-edited away from the spec, so reviewers read the spec and the build polices the implementation against it. With hand-rolled Rust you're reviewing the impl directly — which isn't strictly bad, but it does potentially insert the human back into the loop between LLM iterations.
3. AI in the loop. If you're hand-writing the constructors carefully yourself, the argument for shengen is fairly weak. If an LLM is writing them, "declarative spec + codegen" is, in my experience, a stronger prompt than "describe the constructor in precise English." That's the frame the post is really written for; the Rust newtype point is well-taken outside it.
I'm not saying everyone writing software (w/ LLMs or not) needs shengen / shen-backpressure. The underlying principles it's trying to help you use aren't new at all. But, if you want to get more out of LLMs (esp. multi-turn loops), you probably need something that is deterministic and structured to provide that backpressure context you want to the LLM as it iterates.
You're right on ratios, but actually the ratio is much worse than 6:1 since they are MoEs. The 20B has 3.6B active, and the 120B has only 5.1B active, only about 40% more!
This is neat! I think in general there are really deep connections between semantically meaningful diffs (across modalities) and supervision of AI models. You might imagine a human-in-the-loop workflow where the human makes edits to a particular generation and then those edits are used as supervision for a future implementation of that thing. We did some related work here: https://www.tensorzero.com/blog/automatically-evaluating-ai-... on the coding use case but I'm interested in all the different approaches to the problem and especially on less structured domains.
Looks very buttoned up. My local project has some features tuned for my explicit agent flows however (built directly into my inference engine), so can't really jump ship just yet.
I definitely see different prompts based on what I'm doing in the app. As we mentioned there are different prompts for if you're asking questions, doing Cmd-K edits, working in the shell, etc. I'd also imagine that they customize the prompt by model (unobserved here, but we can also customize per-model using TensorZero and A/B test).
wireshark would work for seeing the requests from the desktop app to Cursor’s servers (which make the actual LLM requests). But if you’re interested in what the actual requests to LLMs look like from Cursor’s servers you have to set something like this up. Plus, this lets us modify the request and A/B test variations!
Sorry, can you explain this a bit more? Either you're putting something between your desktop to the server (in which case Wireshark would work) or you're putting something between Cursor's infrastructure and their LLM provider, in which case, how?
we're doing the latter! Cursor lets you configure the OpenAI base URL so we were able to have Cursor call Ngrok -> Nginx (for auth) -> TensorZero -> LLMs. We explain in detail in the blog post.
We're working on an OSS industrial-grade version of this at TensorZero but there's a long way to go. I think the easiest out of the box solution today is probably OpenAI RFT but that's a partial solve with substantial vendor lock-in.
This is very neat work! Will be interested in how they make this sort of thing available to the public but it is clear from some of the results they mention that search + LLM is one path to the production of net-new knowledge from AI systems.
reply