To be fair, Deno and Bun's enhancements seemed to do a lot to set a fire under Node and have it implement various things it didn't have when both projects launched. This is a good thing.
A bit frivolous, but I once used a spare VPS like that as a one-listener Internet radio station for my mom which worked with her portable radio (she has no interest in using Spotify etc.)
Smaller models have been able to do these sorts of tasks, but a little slower, for a while now. Give a small Qwen 3.8 model a classification task and force a structured output, and it'll do a good job. I've used Qwen 0.8b for basic image classification in <500ms on my local machine for a while now.
There are a few technical details that can reduce the latency significantly (covered in the post) but the real insight has been from watching the reaction to Jev and seeing that there's enough of a market interest to offer it as a distinct thing. The underlying concept/approach was already there.
Not just structured output. Dropping down to logprobs, prompting the model to emit one word as the answer, and then ranking the output tokens to pick your answer works great on small Qwen & Gemma models.
The fascinating part to me is that Jev seems like this technique plus post-training to get multiple independent confidence values for each possible answer.
You can also go beyond Jev. Qwen 3.5 0.8B is fantastic at basic image classification/question answering (including OCR elements) also. Though rather than looking at logits, I get it to output a structured JSON object and it does simple object classification tasks on a Mac at under 500ms a pop (I forget how far, but I think it's like ~250ms) with good accuracy (depending on task).
I've had similar thoughts! I grew up around a heavy smoker. My dad told me to never smoke and I never have. But I randomly tried a nicotine lozenge in my 40s and found it such a "lightbulb on" moment that I've been using pouches for a few years now and my executive functioning still feels better for it (though not as much now as in the first two years, alas).
I had a similar thing with Xcode the other day. I unticked the "download a code assistance model" thing, clicked to continue and it started downloading it anyway (which, to be fair, you can just X out of).
There might be some practical applications of this sort of idea like in situations where you want/have a heavily restricted vocabulary to build from. You can already do this with LLMs but they can get very... "distressed" if you force logits, whereas this would not.
Come to think of it, I'm now curious if it would do well at building SQL queries, say. (30 minutes later: I tried it, and it can do it reasonably well, but a normal model and linting will outperform it, though the inherent guard rails of a limited vocab are still intriguing.)
Almost certainly not. The legislation says: "An operating air carrier shall not be obliged to pay compensation [...], if it can prove that the cancellation is caused by extraordinary circumstances which could not have been avoided even if all reasonable measures had been taken."
That gets them off the hook for even second order effects. No harm throwing in a claim though, there's no penalty for having a claim turned down.
FWIW, on my Mac Studio I get ~24-27 tok/s generation between 0-16k context in - that's on the Q6_K GGUF with speculative decoding on. I have spent zero effort optimizing/improving this so far but will be trying the 4 bit MLX next (I've tended to find models drop off somewhat below 6 bit but maybe that isn't the case nowadays).
reply