Hacker Newsnew | past | comments | ask | show | jobs | submit | kvcm's commentslogin

I haven't gotten around to it but hear it's faster than Strata, their README mentions a "probe" that'll provide actual numbers. Strata supports concurrent requests also, and manages the KV Cache / scheduler very well (inc swapping overflowing parts of it to CPU RAM, though if max context exceeds VRAM concurrency becomes swapping sequential behaviour).

Switch to IQ2_XS IMO: I've seen +50% performance and saves a massive 9GB VRAM, which buys turning on `parallel: 3` (I have "Oh-My-Pi" configured to snap compact after 131000, max length 255000, so I can use the extended context slash command of I need a longer context in one session, on demand. Benches equal to the very highly regarded GSQ RCO IQ3_S (which is 9GB larger). Since you have a laptop 5090, the performance increase will likely exceed what I've seen with a desktop 5090.

The undocumented Oh-My-Pi setting (omp doesn't have documentation rn): ```yaml compaction: modelThresholds: "someModel/*": "f131000" # or "50%" for a percentage of the window ```


Gonna be a lot of people thinking about that models favour the number 7, and they do the same when recommending an option too ("Who’s Aligned to Whom? How AI Agents Shape Experts’ Research Judgment Calls"), and what happens if staff are influenced by what some model opines. I'm having some work done on my apartment, the best models have been badly wrong on most things an actual expert can work out. Similarly, work I've seen on an inference engine, done by experts versus vibe coders.


I had Hermes managing mine, and it made a partial prepayment to help smooth out the bump in my account balance. Unfortunately Billing Support say my $17.4B refund may take up to 10 calendar days to be processed.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: