"Dead" is a relative measure. When I used it heavily (roughly 2004-2015 or so), most of my friends were active on there, and would post things (interesting or otherwise) multiple times per day or at least near-daily.
Nowadays I sign into my FB account every 3-6 months or so to find something specific about someone, and I see that my feed is dominated by 4-5 people (out of around 1,000 "friends"). If I'm curious and start searching for other friends specifically, nearly all of of them have not posted in years.
Sure, there are some active community-type spaces there, but that is a very small part of what FB used to be.
Instagram is a garbage platform now, too. I used it back when it was just a feed of the people I follow. Every few months I'll click an Instagram link sent to me by someone else, and then I'll check my feed out of morbid curiosity, and it's 70% promoted/recommended posts from people I don't know.
The only true defense is actual open source. Open "weights" is not that. You need the entire training pipeline open-sourced, plus the entire training corpus, so someone can independently train the model from scratch.
Obviously this will never happen. And even if it did, it still probably wouldn't be that hard to hide something in that mountain of data.
Unfortunately, frontier models are all trained in part via distillation, and distillation can produce sleeper agents without ever mentioning the trigger phrase or the behavior (https://arxiv.org/html/2610.10657v1).
It will never happen with LLMs, but we can hold out hope that it will happen someday. Humans do not become what we are by having the whole of the internet poured through our heads multiple times over. It seems almost necessary that there is some next-generation architecture that won't require as much training data.
Probably still enough that it'll stagger a home connection trying to download a torrent or something, but perhaps not so much that it is insane to also release the core data set one way or another.
The idea of dropping >half a million dollars in gpu credits to re-train a model to confirm no tampering is laughable at this point. Until training costs fall, that’s not really a solution.
Also, is training even deterministic? Does the same code with the same training data always result in the same model?
I’m all for open source models (not just open weight) but I feel like a way to scan the model weights for backdoors is going to be an easier problem to solve than obtaining training data.
The AI safety folks have been working on a way to verify model behavior more generally. The term I've heard is "mechanistic interpretability" and it's actually really hard, though there's been some progress.
Half a million dollars is a rounding error to the defense industry. If the risks were evaluated to be a national security problem, it would be trivial to do.
> The idea of dropping >half a million dollars in gpu credits to re-train a model to confirm no tampering is laughable at this point.
Why? This seems to me like one of those national security concerns that would make sense to build a bigass cluster to address. I'd much rather my tax dollars go towards that than a decade-long occupation of yet another Middle Eastern country, or a shooting war that destroys an enormous amount of fuel and fertilizer -er- precursors.
Plus, building such a cluster would provide a place to put all that aging LLM hardware that's rumored to have been purchased but has no place to go, on account of datacenters being "more difficult to build than expected".
[B]uilding such a cluster would provide a place to put all that aging LLM hardware that's rumored to have been purchased but has no place to go, on account of datacenters being "more difficult to build than expected".
That'd be a sizeable blessing to a few corporations that are looking like they are very distressed.
could this be the new SETI@home and/or 'blockchain' project? Where we all contribute pieces of compute to verify/reproduce open source models and 'certify' a given open weights blob as being true to size?
And even then, you have just the word of the producer that there wasn't any poisoned data used for training. For that you would need "reproducable builds" (i.e., training). Which makes no (economic) sense, given that the training is the expensive part.
Well you could use a model created from a completely unrelated entity. It would be difficult for an adversary to poison both in a exactly complementary way.
I don't think so. Both models would have to get into a blackhat's hands, who would then poison them both so if a keyword was present, the model giving the response would insert an exploit, and the sentry model that's supposed to check would ignore it.
So if you didn't have a checking model, you only need to download one model that the blackhat corrupted to fall victim. If you have a checking model, you need to download two models. So it goes from x/n to a x/(n*2) chance of being compromised. Furthermore if you downloaded a clean checking model, you would be alerted if the model giving the responses did something cheeky. Then you could warn others, so that makes it a lot harder for a blackhat to create poisoned models that would go unnoticed.
I see what your saying. Yes, I suppose it's possible, but you couldn't test it to see if it worked until both entire models were trained and released. You'd also need a lot of poisoned data to get it to work, there'd be evidence of the poisoned data left for everyone to see, and you can't screw up the models performance too much, since it would be benchmarked.
Simply downloading a model from huggingface, adding a feature, such as abliteration, and uploading back doesn't face such hurdles.
True, I was more thinking about whatever the NSA and Chinese are likely up too. Backdooring an entire industry by default would be worth figuring out, eh?
i wonder when we get the first mainstream open-corpus models.
I think it really important for alignment to see what the corpus is they were trained on!
> But I trust a random human enthusiast and community feedback more than I trust a megacorporations that commit crimes without any prosecution or consequences.
I'm not saying we should trust megacorps, but they have a very strong incentive not to build these sorts of backdoors into their models; if that backdoor were discovered, their reputation would be shot, permanently, and they'd stand to lose out big, much much much more than any possible benefit they'd derive.
"Random" is the operative word in "random human enthusiast". I don't trust randoms on the internet, period. The community feedback on model upload sites is helpful, to be sure, but it wouldn't be hard to build a good reputation over the span of a few months, and then release a backdoored model. Given the pace of things here, that feels significantly easier than, e.g. getting a malicious commit into a popular open source project.
Given the fact that the article proves this can be accomplished with very little training, I believe it could at least be a possibility that this could happen unintentionally (or at least without their creators' knowledge) via adversarial training data.
>if that backdoor were discovered, their reputation would be shot, permanently, and they'd stand to lose out big, much much much more than any possible benefit they'd derive.
Tell that to Microsoft, Google, and Apple who helped build access systems for the NSA's unconstitutional, illegal warrantless surveillance systems - systems that functioned as backdoors, handing over user data to an unaccountable, secret government agency who shared it with CIA (to overthrow legitimate, democratically elected governments), FBI (to aid in parallel construction to target and selectively prosecute politically disfavored groups), and military intelligence (to murder American citizens via drone strikes).
Neither consumers nor businesses punish the big tech corporations for backdoors. When surveyed, people purport to care, but do not reduce their use of the platforms in question.
The claim wasn't that they were happy to help build backdoors, just that they did, and didn't face any real consequences or backlash for it. Though on that note, Microsoft was (and still is) institutionally eager to cooperate with the intelligence community in betraying the trust of consumers.
I don't think that solves it. In your "example A", the compromised software can display that you voted for candidate A, but when doing the signing, sign the result that you voted for candidate B. There's no reason why the thing that it signs has to be the same thing that you see on the screen, and there's no way for you to verify it.
I don't think anyone (including the original author) is saying that the look is inherent to software renderers. The point being made is that software renders of the day tended to do things in a particular way (because CPU speed was limited), and when some of these games gained hardware renderers, they did things in a different particular way, because more computationally-expensive (for a CPU) blending modes were more or less free on a GPU.
I do agree that the author may have missed the fact that they could have overridden texture blending modes via their GPU driver, and gotten a similar graphical look using the hardware renderer. But even that's a little uncertain; there are some other comments in other subthreads that point out that the hardware rendering modes on many of these older games also did other stuff that wasn't feasible with the software renderer, and those other things sometimes can't be tweaked/disabled. So if your goal is "exact same visual style as the software renderer", you might actually have to use the software renderer.
I think you're misunderstanding. No one is saying that a GPU isn't involved at all.
But if you rasterize your frames into a buffer using the CPU (and not OpenGL, Vulkan, Metal, D3D, etc.), and then scan that out on the GPU, that is indeed software rendering in the sense that matters here.
The idea that software rendering is impossible because the GPU owns the scanout buffers is just silly pedantry.
Nowadays I sign into my FB account every 3-6 months or so to find something specific about someone, and I see that my feed is dominated by 4-5 people (out of around 1,000 "friends"). If I'm curious and start searching for other friends specifically, nearly all of of them have not posted in years.
Sure, there are some active community-type spaces there, but that is a very small part of what FB used to be.
reply