You would think, but My 2019 MacBook can barely run an older xcode that doesn't emulate newer phones / tablets.
Some of these responses to my above post are a bit haughty. I'm just reporting from the trenches that the Apple tax is real, not everyone can afford to keep paying up, and a 20% cost increase is huge.
Serious question: do you think the NSA aren't training their own LLMs? (With or without Anthropic and OpenAI's help)
It's a perfect technology for their uses, they get a big chunk of a $100 billion black budget, and they've had access to the research for at least as long as we have.
I can't say what they're doing now because I worked for the NSA 15 years ago but the view of them as an omnipotent power is a product of Hollywood. The government is good at throwing an ungodly amount of resources at something to get a result through sheer attrition, and so they are often the source of original development of technologies. The private sector has always been much better at building a technology to greater sophistication and efficiency. There may be blue badgers in Fort Meade trying to train models but there is no chance they are competitive with the frontier AI companies. It's like saying the government has an amazing home-grown fighter aircraft that is beyond what Lockheed has ever made...they delegate that stuff to private companies for a reason.
Blue badges were for government employees (like I was), and green badges were for private contractors. And yes they have a lot of math and physics guys; my own physics lecturer was in my orientation class, actually. He was there for quantum computing, which reinforces my point. The government can be good at pioneering unproven / uncommercialized technologies, but in general they are like a blunt weapon; the profit motive and lack of bureaucracy eventually makes the private sector far better for improving the technology later. In the case of LLMs, they didn't even originate in government, and I don't think there's any chance they are being developed there at a more advanced level.
Crypto and AI are deeply connected, and you see similar structures/problems in both. Shannon, the “Father (or whatever) of AI”, worked for the NSA and published many papers there that were later declassified.
Here is a banger quote on this by Shannon’s boy Warren Weaver, keeping in mind LLMs came from translation problems:
“One naturally wonders if the problem of translation could conceivably be treated as a problem in cryptography. When I look at an article in Russian, I say: 'This is really written in English, but it has been coded in some strange symbols. I will now proceed to decode.”
> Serious question: do you think the NSA aren't training their own LLMs?
Given the evergreen discussion of "are these companies making a profit"*, I think any LLMs that the NSA (or any other government agency worldwide) may be making are quite far from the leading edge.
* Person A: "they are making a loss!" Person B: "Only if you count training, they make a profit on inference, look at what it costs to run comparable open models on generic cloud servers" A: "Sure, but if they don't train new models they'll be left behind, so they're still making a loss"
That and the way compute is now measured in GW, I think even random low budget vloggers just getting started would be able to spot if the NSA was doing anything significant just from the extra heat emissions or power plants getting built.
The rate of inference compute to training compute is ~10:1, for popular frontier models. Models are routinely overtrained past the Chinchilla optimum now because it makes an immense amount of economic sense to do so.
Worse the more niche and unused your models get, but when this "making a loss" fuckery pops up, it's usually about the big guys like Anthropic, OpenAI, GDM and maybe xAI and Meta. Of which only the latter can be accused of not selling enough inference to offset the training runs.
The real money sinks are: R&D and infrastructure buildouts.
I don't think there is much overlap between people capable of building cutting edge LLM's and the people who want to build a cutting edge LLM for the government.
The NSA managed to deliberately insert a backdoor into elliptic-curve cryptography right under the noses of everyone capable of making elliptic-curve cryptography.
Mathematicians in academia are paid a little less than AI researchers. Companies are willing to pay billions to steal the few people capable of driving development of frontier LLMs from each other. Cryptographers don't quite enjoy the same popularity.
> The NSA managed to deliberately insert a backdoor into elliptic-curve cryptography right under the noses of everyone capable of making elliptic-curve cryptography.
That sort of proves the opposite point, assuming you're referring to Dual EC DRBG, because the flaw was noticed very early on, by people who weren't even involved in its development.
The NSA is government agency. They are certainly not training any world class LLMs. They probably have some specialized fine tunings of existing models, but that's it. They don't have the capacity.
Serious question, do you realize that the NSA are mere mortals? Do you realize how much it takes to train a model? Does the NSA make their own chips or planes? The NSA buys a lot of technology because they can't make their own.
You mean "Rhetorical question," and I didn't need patronising.
They have at least one pretty vast, largely classified data centre in Utah, with a sizeable chunk of the black budget and they also have pretty large data sets.
NSA has had their own supercomputing program for decades. they design and produce their own large scale machines. chips, fabrics, arithmetic units, all of it. they also employ quite a number of hardcore mathematicians, computer scientists, and systems wranglers. if they decided it was of strategic importance there is absolutely no reason they couldn't train their own models.
I guess we're just conspiracy theorists for landing at the objective conclusion that three letter government agencies:
- find "modern AI" to have strategic importance
- have ways to spend loads of money while having a front-facing budget on the record
- could be running a PR program to have Americans think they "buy" access to models like they do, but the AI companies were taken over by these agencies long ago
Look at Google, Microsoft...Apple got away with it by having as much on-device operation as possible so they could wash their hands, honestly saying "We don't have it."
This is the world's largest data gathering operation. Remember after 9/11 when the NSA copied as much Internet back bone traffic as they could?
I'm not for or against, even as a resident, but we certainly shouldn't be naive.
as someone who actually worked at the NSA pointed out earlier in this thread, they have plenty of resources, but also plenty of politics and some execution problems. so I wouldn't put money on them making a great model, but to say that they are completely incapable of doing anything is probably quite wrong.
the issue here that is a forgone conclusion, regardless of where the model comes from and which chips it runs on, is that now they can reasonably comb through all the stuff that they've been collecting. that's a pretty huge operational change.
Even if AI is the next combustion engine, would you bet the farm on a C tier buggy whip manufacturer like Oracle being the next Ford or General Motors?
Why are you asking the author about that? It's rather clear they are just trying to present Oracle's perspective, not whether Oracle will be the one who will champion the next technological revolution.
Google's AI is hamstrung by a culture of safetyisim, by that I mean going beyond what we can all recognise as safe limits to protect the user from imaginary ephemeral things like cultural harms.
So maximal safety at all costs is in itself a cost. They can spend billions on AI but that spend is down the toilet if the user bounces because the AI's persona is a relentless politically correct scold.
People forgot, but Google had their own internal version of ChatGPT before OpenAI. But they never even intended to launch it. If OpenAI hadn't just thrown the technology out there for everyone to see, Google would probably still be sitting on it. Google does tons of original stuff, but they haven't released any original product in more than a decade. All they do now is play catch-up once they see people actually like something.
There's also the major problem of people expecting Google to be right when it tells them something but OpenAI had no starting reputation so it was okay to say "be aware it might be wrong sometimes"
These numbers show that OpenAI is boned. They have no path to profitability and if they raise prices or cut services they will strangle their golden goose.
They could have existed indefinitely as a service layer that was reliant on other companies feeling charitable, like Firefox, but they also wanted to get rich.
It's possible that I'm just not up to date with current news, but I'm having trouble connecting this quote to the article. Or really even understanding the quote at all. Can you elaborate?
The commenter above seems to be describing late stage capitalism, where businesses exist mainly to milk investors, as told by bad boy tech executive Dick Jones in the 1980's action movie RoboCop.
That would imply that the EU has the ability to build its own, competitive AI ecosystem. At the moment, it's mostly just Mistral, and they have been way behind SOTA for a while now.
You can't just legislate this into existence, you also need the money and talent to do it, not to mention the hardware.
Hopefully, EU-based Yann LeCunn's AMI Labs will develop foundational world models at some point. As I see it, the main problem in EU is not lack of talent: it's lack of investments. Mistral itself recently secured 4B, which is 50 times less than what it could have made in the US.
Investing in the EU is like burning cash. Excessive regulation, high costs and taxes (especially of human labor), and investors get punished for creating and growing companies.
Even if you overcome all of this and become successful, you'll get chased by politicians for having too much money (which is not allowed in the EU).
And even if you don't, today, it will be tomorrow.
Easier to just take a long haul flight to your favorite US coast and do it there.
However true that is, it now has only to compete with the US, where any model could be shut down by the Government on a whim with no clear rules at any time.
The rest who are not chased are usually deeply intertwined with politicians, and are running corruption schemes along them.
I'm not one to hate rich people (I'm a capitalist myself), but if any billionaires are to be looked at with skepticism, it's EU billionaires with significant political ties.
You _can_ legislate it _out_ of existence though. The competence is there, no doubt. Just not the complete disregard for copyright law that is step 1 of training an LLM. Rules for thee but not for me, classic colonialism. The EU is so intertwined with the US that even Trump isn't enough to force a clean break.
>Just not the complete disregard for copyright law that is step 1 of training an LLM.
I guess you forgot that EU's unicorn champion, Spotify, started by distributing pirated music they stole off the torrents before getting the rights to that music.
Except they could have killed spotify legally if they wanted to. Their luck was that Stockholm was the capital of the music recording industry so they had easy lines to negotiate with record label execs face to face to not get buried by the Swedish, EU and US laws they broke and would have sent them to jail.
Same thing will happen with the LLMs. Publishing companies will cut a deal with the LLM companies to get a cut off the books and IP they used for the training data.
So this is a very poor cope/excuse of why the EU doesn't have cutting edge models, because it's illegal to steal IP in Europe vs US.
reply