The 'Elbow method' [1], yeah. Sweep a bunch of different k values and pick the one where your target metric starts having diminishing returns. There are a lot of heuristics you could use.
There's still not one particularly great way to pick k for this type of clustering. There kind of can't be as clustering is sort of inherently subjective.
Other methods like HDBSCAN can find the number of clusters themselves. There's still a 'one key parameter' you have to pick yourself, but it can be more intuitive and less sensitive for some datasets, when you don't know how many clusters there are / how many you want.
Sometimes, picking k exactly is useful for practical reasons too. Like fixing the size of your codebook. Or partitioning a dataset into k equal-ish pieces, say to spread across a pool of servers of a known/fixed size.
Yeah, it's not a well-defined problem, so there's no one true solution. Then only question left is "well what do you want?". Always have to pick something.
At the risk of oversimplifying, it seems the field looked at neuron activations and couldn't find the pattern, so a series of convoluted hypothesis were proposed to explain it away. When the actual answer was the pattern was very simple they just missed it. Little bit embarrassing really.
The good news I guess is the actual patterns make a lot of sense with what you'd expect the neurons to do, and don't require nearly as much 'black magic who knows what wonders scale can create'.
Anyone remember Universal Transformers paper (Dehghani et al) from back in 2018? Recurrent transformers have a history as long as transformers themselves.
Somewhat unclear how particularly novel this is vs a way to save compute.
Iroh's discovery is genuinely neat, and node-IDs-as-public-keys is similar to bitbang.
A browser can't join a DHT though because of no UDP sockets, so the connecting side would need a native client, which is the thing bitbang avoids. And DHT lets you locate the peer, but you still need somewhere to trade SDP and ICE candidates, so the rendezvous stays either way I think.
What I can do is keep the broker small enough that self-hosting is realistic, which is where it is now -- no accounts, no registry, nothing that grants access, only a simple broker.
This tracks with what I’ve heard around as well. What changed in the financial instruments?
My understanding is a lot of the loans have gone PIK or otherwise essentially aren’t serviceable at current prices. Do you think that’s resolvable somehow or just lagging implosion?
Nothing changed in financial instruments - we just don't usually have a lot of office tenants decide to walk away like we have with the advent of remote work. The loans will work themselves out. We've seen a couple of buildings trade for way less than pre-Covid numbers, and we'll see more.
In another 10 years downtown Seattle will be aligned with the rest of the market again.
Anisotropy in word embeddings dates back to at least 2017 with word2vec - where there were zero layers.
The cone-shaped anisotropy in transformers is known since at least Gao et al. 2019. That lineage explained it fairly intuitively as an artifact of word frequency and softmax geometry (so a training dynamic).
A variety of papers followed up by adding post-hoc ‘whitening’ steps (from classical statistics/NLP), then adding regularizers to the loss to penalize the anisotropy, eventually penalizing the covariance matrix (a la VICReg), and then the SIGReg method as a computationally much cheaper way to approximate the full covariance.
As another commenter pointed out it’s also similar to the InfoNCE/contrastive learning objectives. Where terms to increase uniformity (spread out evenly) on the hyper sphere were added. Like the SimCSE (Gao 2021) paper or the excellent alignment/uniformity breakdown from Wang & Isola 2020.
This proposed dispersion loss seems to be similar in that it pushes things apart by penalizing cosine similarity. Although this one works on the tokens within one sequence. Usually contrastive methods mean pool the sequences and then contrast against the other pooled sequences in the batch.
There's still not one particularly great way to pick k for this type of clustering. There kind of can't be as clustering is sort of inherently subjective.
Other methods like HDBSCAN can find the number of clusters themselves. There's still a 'one key parameter' you have to pick yourself, but it can be more intuitive and less sensitive for some datasets, when you don't know how many clusters there are / how many you want.
Sometimes, picking k exactly is useful for practical reasons too. Like fixing the size of your codebook. Or partitioning a dataset into k equal-ish pieces, say to spread across a pool of servers of a known/fixed size.
[1] https://en.wikipedia.org/wiki/Elbow_method_(clustering)
reply