And the 'better' ones are slower and cost more for a tiny bit more accuracy. Its hard to sell that as being better when speed and price have been JEVs main selling points.
I have roughly the same workflow, also with plannotator - which I like a lot - and haven't used or felt the need to use _plan mode_ for at least 3-4 months.
Then telling Claude to work on a document, the instruction is kept to its core.
Now when bcherny explicitly mentioned that it merely adds a single line - it explains why I don't need it.
What may be concerning about "super plan" mode from the creators (or a skill, for that matter) - is that tuning the amount of effort, and how much deep to dig - may become too hard, as it will interfere with several embedded paragraphs explaining what to do, how to do, where to do, etc'.
What I do look for is even better plannotator ability to track changes, combining historical comments (like Google docs), and git blame of several "generations" before current reviewed doc.
While possible to commit, I'd still loose the comments themselves, and the commit will be at least per changeset, and not connected ro a specific changed section.
Roughly speaking, I'd be happy if plannotator would persist something similar to github PR reviews combined with Google docs comments & suggestions.
I think the meaning of can't hallucinate in this model is that the type won't be hallucinated.
So if the generated schema is for a tool call for calculator, then the numbers will be valid numbers for sure (and not random words).
To me, it looks similar to BNF schema already introduced and implemented few years ago: generally speaking - it limits the next token that is allowed to be generated, probs are drawn from a subset tokens.
(tbh, I'm not sure why it didn't pick up as a more standard interface to LLMs, as it made a lot of sense back then, and now.)
Yeah, I thought about constrained generation as well. I've actually done something similar with local models before. And you can even get a "confidence" score by looking at the logits (something along the lines of logprob("YES") + logprob("Yes") + logprob("yes") - logprob("NO")...
There's also a cheeky "one of the models hallucinated a link" in the wiki jump example that most likely could have been avoided by properly using grammars. You can setup constrained gen so that only valid options (say from a list) can be outputted. Their own inference lib likely does that. So comparing to one that doesn't is a bit cheeky.
That being said, after a brief look at the site I could see this working. Especially if this can be ran locally, the speed and cost can enable some workflows where you have this as an "overseer" layer over say a cli agent. After each step you run through a list of "questions" ("is the task completed?" -> yes -> "does the edit touch files it shouldn't" / "does the edit follow our code writing policies") etc.
edit: extra points if the "question" rubric is also generated by a higher abstraction model. Say "/goal Build out auth" -> generate_rubrics(goal) -> "Is auth implemented on all endpoints" / "Has code touched anything else than auth" / "is this following the best practices" / ...
AFAICT it is the same interface as you describe, but the underlying inference algorithm is fundamentally different, hence the speed gains. There is an application I am currently working on right now where this typed output predictor is the performance bottleneck. I'd be very interested to see how this performs.
It did, no? If you send a JSON schema to an LLM endpoint, at least for OpenAI and Anthropic, they turn it into a grammar behind the scenes and use it for constrained decoding.
Worse, if you watch the GN video this article is based on you'll find the TV can figure out how to leverage other non obvious networks (threads, etc) to reach LG servers. Worth a watch to see how bad this is: https://www.youtube.com/watch?v=6IFVTcM28KA
So - in theory - something like a massive chat client, discord like, can be implemented via this solution?
And what would be the pricing of such a solution.
Cheap-serverless-discord
Thank you! I spent a decent chunk of time designing PicoMQ, so the documentation had a natural progression.
Exactly, streams can essentially be rooms, and since the ordering is preserved, a Discord-like application is a strong use case. I’m even considering building one using PicoMQ as an example showcase.
The pricing is going to be dirt cheap, and the best part is how easy it is to scale up vertically or add nodes. For some raw numbers, assuming 1M messages/day, 200B per message, ~6GB/month, all-inclusive, it would be $30 to $150 a month, and storage would be the cheapest part.
Yeah, much of the base cost I mentioned was from the compute, networking and S3 write costs, which could sustain a lot more messages for sure. It also depends on the number of active streams/rooms, the throughput bursts, as opposed to a steady state.
But as we speak, I am in the process of running open-benchmarks on AWS with cost attribution. I'll be posting them in the docs with transparency soon enough.
Read and write through cache already exists today, which would work in favour of both cost, latency, and consumer reads fanout.
I like the HN Mirror over a Discord-like app for simplicity. No doubt that's where my weekend is going!
> They may only have bought themselves 6 more months of time given their purported burn rate
If they had only ~6 more months they (+auditor) had to issue a warning.
The 6 is not a hard number, AFAIK, but surely a point where it must be reported.
That was indeed my intent. Pre-IPO, SpaceX already had _some_ cash in hand, and was burning >4b per quarter IIRC; so presumably _some_ runway. That said, they also got the anthropic/openai monthly payments coming in soon (already started maybe?). The next earnings release will be the interesting one IMO.
Historically, vehicles actually did have completely interchangable motive components. They were called horses.
It would have been difficult given the state of other technology at the time for the inventors of the internal combustion engine to have supplied it as a drop-in horse replacement for your carriage, but you could kind of imagine that working with current technology.
I don't like the impulse to regulate, especially as a formative measure. Regulating new technology before industry has time to experiment seems like a recipe for high prices, few choices, and low competition.
