Google "Huawei Nortel Networks". The fact is that Nortel has an enormous volume of R&D stolen from them and the allegation is that Chinese telecom providers were the main beneficiaries.
Edited: To reply to a child post, this is about the allegations referred to in the parent, not a statement of what I think is true now
My mom worked at Nortel in the 1990s at their Ottawa R&D campus. As a teenager visiting, I asked about corporate espionage, hoping for a story about Cisco or something. Every one of her co-workers just belted out "China" at the same time. It was apparently an open secret that the Chinese were hoovering up IP and management didn't care. It's a bit of a shame as Nortel's telecom kit was considered top-notch (especially the cellular stuff), but it gave the Chinese an enormous amount of help catching up.
That being said, the Chinese then went off and innovated off of it and Nortel imploded, mostly due to other poor management reasons. My mom was at least smart enough to cash out most of her equity as she got them, though. Lotta people lost everything.
The patents they built today are founded on the patents they derived stealing from nortel. Qualcomm needs these whitewashed patents as part of this frand deal.
Nortel's patents were among the first that were critical to modern MIMO OFDM (they really were on to the future in that sense, it's not hyperbole), but most never went to Qualcomm or Huawei. When Nortel went belly-up a consortium called Rockstar co-owned by Microsoft, RIM and a few others ended up buying them for a hefty price (more than $4 billion). It's possible some of them have made their way over to Qualcomm or Huawei as part of sales over the years I guess.
Huawei's patents aren't nortel's patents. The patents huawei generated through their own work are based of off and substantially informed by the stolen work nortel did that nortel patented. It doesn't have anything to do with nortel's actual patents. It has to do with the prior art that was stolen by the prc and given to huawei.
I wonder if the NCD metric says something about distillation too. Would you expect that a model that has been distilled/seen traces from other models would have a smaller NCD? It would be really interesting to see if this holds up and provides evidence of distillation or certainly evidence of model outputs being used in the training mix.
Hello! I wrote the above article and did the NCD on model outputs. The same thought crossed my mind when I saw Gemma misclassified as Gemini quite frequently. And GLM almost as Claude and not as Gemini at all. Gave me the feeling as if GLM didn't train on Gemini generated synthetic data at all but mainly on Claude and GPT.
Here’s my understanding. It is intimidating to write a response to you, because you have an exceptionally clear writing style. I hope the more knowledgable HN crowd will correct any errors in fact or presentation style below.
Old school word embedding models (like Word2Vec) come up with embeddings by using masked word predictions. You can embed a whole sentence by taking the average of all the word embeddings in the sentence.
There are many scenarios where this average fails to distinguish between multiple meanings of a word. For example “fine weather” and “fine hair” both contain “fine” but mean different things.
Transformers are great at producing better embeddings by considering context, and using the words in the rest of the sentence to produce a better representation of each word. BERT is a great model to do this.
The problem is that if you want to use BERT by itself to compute relevance you need to perform a lot of compute per query because you have to concatenate the query and the document vector to produce a long sequence that can then be “embedded” by BERT. Figure 2c in the ColBERT paper [1]
What ColBERT does is to use the fact that BERT can use context from the entire sentence and its attention heads to produce a more nuanced representation of any token in its input. It does this once for all documents in its index. So for example (assuming “fine” was a token) it would embed the “fine” in the sentence “we’re having fine weather today” to a different vector than the fine in “Sarah has fine blond hair”. In ColBERT the size of the output embeddings are usually much smaller than the typical 1024 you might expect from a Word2Vec.
Now, if you have a query, you can do the same and produce token level embeddings for all the tokens in the query.
Once you have these two contextualised embeddings, you can check for the presence of the particular meaning of a word in the document using the dot product. For example the query “which children have fine hair” matches the document “Sarah had fine blond hair” because the token “fine” is used in the exact same context in both the query and the document and should be picked up by the MaxSim operation.
If this is true, my understanding of vanilla token vector embeddings is wrong. my understanding was that the vector embedding was the geometric coordinates of the token in the latent space with respect to the prior distribution. So adding another dimension to make it a "multivector" doesn't (in my mind) seem like it would add much. What am I missing?
I think the important thing is that the first approach to converting complete sentences to an embedding was done by averaging all the embeddings of the tokens in the sentence. What ColBERT does is store the embeddings of all the tokens before then using dot products to identify the most relevant tokens to the query. Another comment in this thread says the same thing in a different way. Feels funny to post a stack exchange reference, but this is a great answer!
I have a theory that organizations that grow fast and scale well all have this “cellular model” at their core.
Investment bank trading desks in the pre-2008 era, partnership at the big strategy consulting firms and even “multi-strategy hedge funds” now are actually all collections of very incentive aligned businesses. They share the Creo quality of making lots of millionaires and people looking back on their time there as one of great freedom and achievement.
In all these places, employees are paid according to the revenue they generate, with seemingly no ceiling to what you can take home.
It is true that the size of any one cell doesn’t scale beyond a small number of people. But all the organisations I mentioned above scale by having units tackling small pieces of vast markets.
The main lesson I took away from reading “Barbarians at the Gate” is that big companies hugely suffer from the principal agent problem, where management is mostly out to enrich themselves at the expense of shareholders and employees (sometimes). This looting is however only possible at a company that was established by a founder with a deep vision and passion for the product and has set up systems and culture that generates sufficient cash for the professional management to leech off.
What I have not read yet is a systematic study of these “cellular organizations” and what the common features are that make them successful. My guess is that the key is that each “unit” or “cell” has measurable economics that makes it possible to share the economic value over a sustained period of time. A bit like why sales people get paid a lot.
Agreed on cellular, but life is not a picture, rather a movie… what works at a stage starts attracting people, and eventually you let the wrong people in. A bit like the hype curve. These wrong people start poisoning internal processes and culture, seeking to cash out. And then the model blows up.
The migration of shitheads from Wall Street 80s 90s to Silicon Valley - technobros is for me a solid example of this.
Indeed, an organism can only survive long enough if it can resist infection. For this, an organization should be openly hostile to certain forms of conduct, and should have a way to expel people who bring in wrong values, especially in management.
This is a really hard problem, due to its influence on the morale, and the danger of weaponization of these mechanisms by bad actors.
I suspect that for most Bloom filters, the most commonly used hash functions are “good enough”. There’s also some literature to suggest that using just 2 hash functions and recombining the results is plenty. See kirsch-mitzenmacher [1] and [2]
Depending on the function you may even be able to use "two for one" hashing, splitting e.g. a single 64 bit hash into two different 32 bit ones and combine those.
reply