You are on the very top of Maslow's hierarchy of needs. Most people never had that luxury. People dying from plague with a life expectancy of 30 were happy for a meal. I'm not trying to invalidate your struggle (sorry if that is how it come across), just offer some perspective. Life is good.
I feel that these potential Yolo side effects can be managed with more control if supervised with a modified prompt.
> When the user's proposed solution is bad, replace it with a better one
When the user's proposed solution can be improved upon or modified due to some critical missing information, suggest a revised plan and prompt the user for how to proceed.
backon has more built-in features (circuit breaker, hedging, rate limiter, generator retry, metrics, trio support, `retry_with()`), zero dependencies, and aims to be faster. tenacity is much more mature (9+ years), has a massive community (7k stars, 30M+ monthly downloads), broader Python version support (3.8+), and is battle-tested in production at scale. backon is the innovation leader; tenacity is the reliability leader.
If in short, for many inference tasks the bottleneck is memory bandwidth. Suppose you have a machine with a memory bandwidth of 256 GB/s, and let's say you want to do inference for 4B model (model with 4 billion parameters). If you will load the model in BF16 format (16 bits), each forward pass (i.e. each token generated) will require roughly ~8 GB of memory bandwidth. So, 256/8 = 32 t/s, and that's the generation speed you will be strictly capped at even if your processing power is measured in exaFLOPS. But let's say now that you have decided to instead quantize the model and then run the quantized version. Suppose you have made a Q4_K_M version (4 bits + some weights will take more). Now each of your forward passes will take roughly 2-3 GB (rough approximations, reality is different) of memory bandwith (actually, it will be around 2 GB), and even in the worst case 256/3 = 85.3, while 256/2 = 128 t/s. Quants can reduce quality of the model and lower it's performance, but in most modern quantization methods those losses are usually negligible (although, of course, they're still present). So, as you can see, it can be concluded that quantization "widens" (it's not removing it fully) memory bottleneck while still preserving (not always though) acceptable quality.
(Sorry for my terrible English, it's not my native language)
So let’s start with a really simple decoder transformer with a single layer and single attention head, and train it to predict the next token in a sequence of text. To predict the next token you need a few things: a query for the very last token in the sequence, and a key and value for every prior token. You take your query and compute a dot product with every prior key (two large vectors in, scaler attention score out). That scaler attention score first goes through softmax, and then becomes the weight you use to compute a weighted average of your values, new value goes through the mlp, mlp output is projected into the logits from which you sample your next token (that’s the general idea at least skipped a few steps).
The last query in the sequence will be new for every new token you predict, but the set of prior keys and values stay the same, ie keys and values are reusable. The key value cache gets bigger and bigger for each new token you add to the sequence, and that’s where compression comes in. You have to store the keys and values in vram, and you’d like to keep the size down by not storing the raw uncompressed tensors. To make this work well your compression needs two things: it needs to be fast so that you can compress and decompress on the fly, and it needs to play well with softmax attention. Prior attempts at compression usually suck at one or the other, either the speed to decompress is too slow and your token/s takes a hit, or you lose important precision and the model output quality suffers. The claim in the paper is that they’ve made progress on both.
So limiting max context length also reduces VRAM needs a bit? If cache is 20% of total, 1/10th of context as a limit would mean 18% total memory reduction.
I can certainly see the appeal of distributing the context with vc. However, I have always imagined this to be integrated into an existing kanban workflow, similar to a Jira or gh issue board. Perhaps agent specific, perhaps not.
Furthermore, an existing kanban (ticket) workflow will expect you to refine the context into something more ... concentrated, or at least something that we are used to seeing as developers working with tickets, at least more so than the chat history that seem to be favored.
Have you put any thought into how this would integrate into such a process?
I did - GitHub and Trello (and I expect Jira) have APIs that could be used to hook up an MCP server. I liked the idea of conversing with the agent in the ticket, but I decided against that because I'd have to keep refreshing the issues, and it seemed a bit janky moving in and out of the IDE.
I also considered a full harness that could stream / sync the responses, but as per my comment below, implementing a full harness meant loosing a lot of the IDE integration features that come with the hand off to GitHub Copilot.
> I went down the route of implementing a full harness for a while like Vibe Kanban, but the issue was that it was unlikely (without significant effort) to be as good as Github Copilot chat, and it meant forfeiting all of the IDE integrations etc (like diff visualisation for the agents actions etc).
Having worked with a flow similar to this for a while - the markdown files become quite valuable as a history of planning and decisions for features. I didn't want to loose that. I just needed some help with managing the plan files I was maintaining - which the kanban board tooling does. A few command shortcuts via @kanban help too
Regarding what goes into the files, the agent tends to be quite concise - you don't see the whole train of thought that some of the harnesses surface.