On its own, each trit doesn't encode much information at all. But it's not about information at the individual level -- it's more about the shape of the network.
I appreciated this comment [0] from earlier in the thread by paul_mk1:
> My best guess is that it is encouraging the network to choose good underlying subnetworks to solve the problem, similar to Lottery Ticket Hypothesis. With ternary weights it is just about who connects to who (ie a graph), and not about the individual weight values anymore.
For myself, I've done a lot of work with image hashing (such as pHash and dHash) -- and in those, you throw away a LOT of information, but simply by keeping the value of each region and tracking whether or not it's above or below the average (essentially, the sign), then it's astounding how robust those algorithms are. Because you don't look at the individual pixels of an image, but it's very good at capturing the impression of the overall _shape_ of the image.
It's less about each individual datum, and more about the shape of the network.
If you're not familiar with Lottery Ticket Hypothesis, that would be worth reading up on.
I appreciated this comment [0] from earlier in the thread by paul_mk1:
> My best guess is that it is encouraging the network to choose good underlying subnetworks to solve the problem, similar to Lottery Ticket Hypothesis. With ternary weights it is just about who connects to who (ie a graph), and not about the individual weight values anymore.
For myself, I've done a lot of work with image hashing (such as pHash and dHash) -- and in those, you throw away a LOT of information, but simply by keeping the value of each region and tracking whether or not it's above or below the average (essentially, the sign), then it's astounding how robust those algorithms are. Because you don't look at the individual pixels of an image, but it's very good at capturing the impression of the overall _shape_ of the image.
It's less about each individual datum, and more about the shape of the network.
If you're not familiar with Lottery Ticket Hypothesis, that would be worth reading up on.
[0]: https://news.ycombinator.com/item?id=39544500