I've seen a graphical designer struggle to use an analogous product. They tried prompts like "remove background from this photo of dancers, don't change their face": the model couldn't do it: it always changed the faces at least a bit. In other words, the model was not removing the background, instead it was generating the whole picture anew, but without the background, which not only seemed inefficient, but was also way too lossy.
The technically correct method to do that would be:
1. generate/create a mask or a depth map
2. apply map to protect regions to keep
3. fill unwanted regions with random noise
4. run the diffusion model
5. (optional) run again on the whole image woth fewer steps if the protected regions aren't blending well
The fact that process like this isn't automated suggests that image generation hasn't caught on - though, tbf, I am among ones that still find generated images magically discomforting and I'm negatively biased.
That section shows a single example, where pixels in some rectangular region around the selected object are being changed. For background removal, that rectangular region would encompass the entire image, so it's not clear whether you have any particular advantage there.
The memory invalidation based on tying memories to pieces of code (e.g. functions) is the big thing here, I think, though I see why it wouldn't be front and center.
I've always found memory systems icky, because it's difficult to tell whether a memory is still valid. Also, discoverability is always a concern (when is a memory relevant). This being both a code-graph and a memory system (seemingly unrelated things), actually addresses this in a novel way.
True. The kind of guarantees you can get when staying within the database can often solve whole categories of problems. But, DX of having non-trivial logic inside Postgres is not great either: I feel that I'm making a significant trade-off. I've also seen interesting languages that compile to SQL, would consider those in some cases.
The process by which benchmarks are setup and run does not correspond at all to how human developers engage with a coding agent. At best it is a loose proxy, and often a bad one.
What benchmarks are usually good at is showing to what degree new models are better than old models. What they are not good at, by construction, is showing that harnesses are well adapted to how people use them.
I presume you're not talking about autonomous agents, right? Because then you could give it a set of tasks and check its success rate. But, even so, are you not giving it tasks? Are you principally interacting with it through discussions that are more difficult to quantify? Even question answering has benchmarks. I'm having trouble imagining how you use them (or "how people use them" in your words).
I don't know if you noticed, but Opus 4.6 was peak for human-computer interaction. Everything has been fairly downhill from there despite better benchmarks, at least in that one regard. Opus 5 and 5.5 are clearly a step above in capabilities than 4.6, and I don't think anyone wants to go back, but 4.7 and 4.8 were arguably worse overall. I genuinely feel I got more done with 4.6 and often switched back, prior to 5 coming out.
Why? Because 4.6 actually talked like a human being. It actually organized its thoughts well, and got the main information across without the wall of text that makes your eyes glaze over. So from the perspective of human-computer interaction and maximizing the productivity of a developer+agent team, 4.7 and 4.8 were regressions. Despite much better benchmark performance.
Even if we consider autonomous agents, that benchmark is not indicative of how well they will interpret *your* requests. Or how well they will interact with other agents in a flock/swarm situation. The benchmark just doesn't cover this. (And the difference can be nontrivial! Sakana AI's published results show two generations of uplifting potential from better harnesses.)
It's not sus at all. You're looking at it from the perspective of a general purpose system that is not adapted to your use case. Basically you prompt the model and then let the agent do everything. That's the use case you have in mind when you think that it's about being "a tier above frontier labs' offerings".
It's missing the point. I mean think about the basics, why open the huge bash hole only then to have to close it? If you think about it logically, the only way you can sandbox bash is by writing your own bash implementation specifically for agentic use cases.
Do you mean, with the naked eye? It needs to be quite dark - minimal light pollution.
Then some satellites look like stars that move relative to the actual stars. It's not too hard to find one with a bit of patience. It's pretty amazing.
You don't need to as much as parse HTML to read it. Building a client by hand in C even should be pretty painless as long as you don't write your JSON parser yourself.
that's fair. thanks for pointing it out, there's a lot of helpful feedback here that will let me optimize this (admittedly) unnecessarily complex project :)
It's interesting to think about this in the context of parent poster's story about the Amiga emulator in the nuclear power plant.
Or, more broadly, in the context of any situation where the software is closely coupled to a specific operating system and/or hardware.
Having the source code would have helped tremendously, but might not have been sufficient because of the hardware story and the regulatory approval factor.
reply