I understand the obsession with low bit quantization, but it is empirically quite evident that it is not possible to compress models to less than 4 bit per weight without severe loss of capabilities².
It may be nice as an experiment, but it is obviously a very inefficient route for model training: spending all the flops on a saturated model only to prune its capabilties.
²As to why, I have seen few explanations. But the empirical evidence is there.
First I just want to say, specifically for Post Training Quantization, I absolutely agree with you. As for why 4 bit is the limit, the ParetoQ paper has some interesting theories: https://arxiv.org/abs/2502.02631
That being said, there's a slight misconception about models at lower than 4bpw. There's no fundamental reason why a transformer with low bit weights would be inherently incapable of doing high dimensional function approximation, but training a model at one precision and then quantizing to a lower precision means the training loss is never calculated based on the quantized state.
There's a huge difference between "I trained a ternary model from scratch to do X" and "I trained a model at fp16 to do X and then squashed the hell out of it". Quantization Aware Training is the solve, but it's really expensive compared to a one-time, offline translation of existing weights.
> There's no fundamental reason why a transformer with low bit weights would be inherently incapable of doing high dimensional function approximation,
Sure, in fact you can train a 1 bit model to approximate any function. But that is not the point. The problem is that the capacity per weight is diminishing below ~4.5bpw.
With QaT, can you compensate for that by adding more weights. So, instead of training a 1B x 4bpw model to capacity saturation you can train a 4B x 1bpw model and achieve the same capacity. But you will end up with the same number of bits in the model, just distributed differently.
I am quite certain that similar behavior is true also for larger models - it may be more difficult to experimentally demonstrate due to information bottlenecks and the flops required to train them to saturation.
Maybe? I am using a 3 bit quant for Qwen Next Flash in a 64G Mac using Sushi and the results for coding, design, refactoring, etc. are very good.
That said, after being satisfied with Poolside.ai’s Laguna XS 2.1 4 bit quant, when I upgraded to a 64G Mac, I now only run the 6 bit quant and the results are better.
Unfortunately, I think that benchmarks are not useful for me, and I need to take time seeing what works for me: this is one reason that when I find a combination of model, harness, etc. that works for me, then I tend to use it for a month or two before looking around for new stuff.
Its not so clear to me what their real explanation is. Yes, it is possible to redistribute the quantization error, but this only works when the model is not trained to capacity.
Influencers have latched onto the pitch that everyday people can run frontier models on an 8gb GPU while sticking it to the labs. There's a large audience of people who haven't had the hardware to test larger quants to see the difference.
They would be better served with smaller models that can reliably call tools and generate structured outputs without looping or totally hallucinating. These projects exist, but aren't getting amplified.
Most popular would be Qwen 9b or Qwen 35b with experts offloaded to CPU. Even these aren't great, but they can be run at higher quants on lower hardware. We'll see what Qwen releases version 4 in the coming weeks.
It is true that block-headed quantization of everything doesn't work. But, as I am sure will read, if you remove the spiky parts of the parameter values, the residue can be dramatically quantized and compressed while retaining performance.
This is a multi-modal sort of compression where you use different techniques for different phenomena. Simply compression all of the weights, each in isolation, ignores the benefits of compressing them collectively.
The capacity of your model scales with total number of bits in your weights. Shannon entropy.
Yes, there are many way to distribute the error. And LLM are usually not trained to capacity or are prevented from this due to information bottlenecks in the gradiant flow. This allows unlocking wasted capacity by distribtion of the quantization error. But I am pretty certain that there is no way to trick Shannon.
I mean, this paper clearly shows that all models severely degrade for low bit quantization. Also, these experiments were performed on relatively old models that are not trained as close to capacity as newer ones.
There are mixed weight models that make it work. I have been using qwen3.8_27B_UD_Q3_K_XL. This does 46 tok/s on a 5080 with 16gb vram and it can pretty much do any systems work because it can test to make sure it did it right.
Is this architecture actually able to generalize or is it mostly based on memorization? Have you tried some basic tasks that require generalization? e.g. number addition etc?
The model is way too small and undertrained to make any generalization claims. I want to wait until it reads the whole corpus I gave and then test it on some simple established benchmarks to see how it will behave.
Seems a bit premature to make an HN post about then, imho.
It's an interesting idea, but it doesn't really do anything interesting yet. I looked at the output in the training run and it is a far, far cry from intelligence. Worse than GPT-2 as it stands.
I do hope it will perform well when scaled and trained, though; best of luck.
>Notably, the neural network component behaves like a compressed index of the training data.
The "Compression" part is indeed about the model finding order in the training data. But this is not about applying some predefined compression algorithm on it, but by actually learning an algorithmic representation of the data.
This has nothing to do with "consulting the training data", because the training data cannot be reconstructed based on this information.
Roughly speaking, there are two types of information being stored in the NN: Memorization and generalization, or, shannon entropy and kolmogorov complexity.
great distinction. can we use it to freely produce and share mp3/h264 copies of any media? because it is provably impossible to reconstruct original based on that information.
But keep in mind: So far there is no way to train the models while completely avoiding memorization and only including generalization. That would be a great way to avoid any copyright issues, but all attempts I have seen so far were fairly limited.
Because I really want to see difference between approximating input signal using smooth orthogonal basis functions, and approximating those using quantization with rate/distortion feedback as "violates copyright, compression algorithm", and then approximating input signal using perceptron, with backprop loss based off original media as "not compression algorithm, skip jail"
And I'm not sure video, images and motion are actually 3 different things. Images are just still motion and videos capture motion, so it's really just "video and audio" of which one is visual, the other is not, thus my confused/surprised comment.
>When Intel finally shipped it at the 45-nanometer node in 2007, Gordon Moore called it the biggest change in transistor technology since the late 1960s. The breakthrough was not the material. It was learning how to process the material at scale.
Its curious that they picked this example. The challenge with HKMG was not the material itself, but how to integrate into into the transistor stack.
There were two completely different approaches: Gate first and replacement gate. Gate first is what the industry was already using for silicon oxide so everybody tried to go with as little change as possible. Only intel decided for replacement gate, which worked much better and reaped some other benefits on the way.
This was a watershed moment in the industry and ultimately led to some of the players dropping out of the cmos race.
But is this really a "scale-up" problem? It required development of novel manufacturing processes (atomic layer deposition), but was still mainly a process integration and device engineering topic.
The part of the thesis I have to agree with is that there is a data problem. The development above relies on executing lots of time consuming and tedious split experiments that often cannot be parallelized. The outcome of this relies heavily on the experience and diligence of the experimenters.
It's probably well suited for an "autoresearch" approach, bridging to the phyiscal world and dealing with the timescale is the challenge.
I had Opus 4.5 design an LLM inference engine in verilog, including firmware and automated verification a while ago: https://github.com/cpldcpu/smollm.c
It's of course far from optical. But lowering the implementation through the abstraction levels turned out to be extremely powerful.
Can you suggest some tutorials for Verilog and FPGAs in general?
I have a spare Tang Nano 9k but I don't feel confident about blindly asking Claude to vibecode me a solution and still would like to have at-least a basic level of understanding.
hm.. has been quite a while for me. The good thing about the Tang Nano is that it is supported by the Yosys open source toolchain. There are quite a few resources on the web when you search for the combination.
Wow! And it also implements a very interesting variant of SUBLEQ that is turing complete.
>This VM implements an OISC - a One Instruction Set Computer. That instruction takes three signed 32-bit operands, a, b and c, and runs a program from memory m[] as follows:
1 PC (program counter) starts at 0
2 Fetch the next instruction (32-bit signed operands a, b and c)
3 If the low bit on any operand is set, remove it, and replace that operand with m[operand] i.e., a dereference of that address
4 Set m[b] = m[b] - m[a]
5 If m[b] is 0 or negative, set the PC to c, otherwise increment PC by 3 words
There's always someone somewhere who, with hindsight, did something that could be retconned into being similar to something important we've got today, von Däniken being an extreme example. Not putting down Losev's work, but accidentally stumbling on an interesting physical effect that you treat as a curiosity and engaging in targeted research to turn in into a product is a very different thing. For example the FET was envisaged multiple times in the same time frame as Losev's work, but wasn't rigorously pursued until Bardeen et al came along.
> He used these junctions to build solid-state versions of amplifiers, oscillators, and TRF and regenerative radio receivers, at frequencies up to 5 MHz, 25 years before the transistor. He even built a superheterodyne receiver.
That one calls them "negative resistance diodes" but I don't see how you can make a functional solid state amplifier and the like without it being a transistor.
The USSR famously invented everything the west did but years or even decades earlier, only for some reason never commercialised any of it, to the point where it became a bit of a running joke like the Su-24 "validating" the design of the F-111 which preceded it by some years. So I'd take any claims like this with a bit of a grain of salt.
Yeah, that pattern can be seen everywhere in semiconductors. E.g. the transistor invention vs. Lilienfeld, Heil, Matare etc. So the scope is more narrow than "Inventend Semiconductors".
Generally, there seems to be a tendency to disregard discoveries from outside the US. I think this pattern can still be observed today...
Other examples: Invention of light bulb, telephone.
What do you mean with "open-source"? Of course, the inference code for all the open weight models is publically available - see llama.cpp or hf transformers.
There are, however, very few models where also the full training pipeline is available. Olmo by AI2 comes to mind.
It may be nice as an experiment, but it is obviously a very inefficient route for model training: spending all the flops on a saturated model only to prune its capabilties.
²As to why, I have seen few explanations. But the empirical evidence is there.
reply