Hacker Newsnew | past | comments | ask | show | jobs | submit | pythongiant's commentslogin

Tenosra is all about making AI more efficient and resourceful you can read more abour it at: https://www.tenosra.com/proxima


Hey guys, i'm especially interested in feedback on the kernel design and integration onto mlx lm.

Its on pypi as well as a simple pip install mlx-turboquant :P


KVBoost is a drop-in replacement for AutoModelForCausalLM. Same API surface (KVBoost.from_pretrained(...), engine.generate(...)), but with cross-request KV reuse, FlashAttention-2, AWQ layer streaming, and speculative decoding bolted on.


Here's the repository incase anyone wants to have a look at the code. leave a star if you find it interesting :P https://github.com/pythongiant/KVBoost


Thanks! this is a weekend project that i am working on in the side just to learn more about ml engineering and custom cuda kernels. didnt think much about the website


Makes sense, fixing that. thanks!


my initial choice was to use Rust for this actually (Probably should've too :P) but i went with python for an initial mvp/skeleton for a future rewrite


KVBoost is a chunk-level KV cache reuse library for HuggingFace models (pip install kvboost). It supports two recompute strategies (selective boundary and CacheBlend), int8/int4 KV quantization for 2–4x RAM reduction, disk-backed cold storage, and 11 architectures including Llama, Qwen, Gemma, Mistral, and Phi. On Qwen2.5-3B we measured 47.9x TTFT speedup on an 8-turn conversation, 21x on code context reuse, 100–743x faster than MLX, and 3–41x faster than vLLM-MLX — including interior chunk reuse where vLLM gets zero hits. Outputs are token-for-token identical to baseline under greedy decoding. Works best on 3B+ models with 500+ token shared context. GitHub: https://github.com/pythongiant/KVBoost


Even the things that should be normal dashes are em-dashes


En-dashes are not em-dashes, and they're standard typography for numeric ranges.

https://en.wikipedia.org/wiki/Dash#Ranges_of_values


I don't get it. The output of the CacheBlend paper is in LMCache. Did you compare against vLLM with LMCache? This is confusing.


slop


just published a beginner-friendly, hands-on guide to GPU programming with CUDA aimed at folks who are comfortable with basic programming but new to parallel computing. Instead of throwing API references and jargon at you, this guide takes you through the why and how of CUDA step by step. It starts from first principles and builds up to real, runnable code. The goal is to make CUDA intuitive. you’ll walk away understanding how your GPU executes work and how to write code that actually runs fast.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: