Another way to view this is as metonymy. "Google" is a metonym for "Google Staff", so using "their" to refer to the humans-making-up-the-corporation (as referenced by the metonym) is appropriate.
I agree it's correct English to say "Google have launched their new product".
If you want to be even safer, you could use a physical network-connected KVM. The only interfaces those expose to the connected computer are HDMI (graphics read only) and USB peripherals (write only). Sometimes they can also physically power off the hosts to which they're connected.
It's a pretty darn slim surface for an attack. Technically doesn't meet the definition of an "air gap", but it gives you 100% of the utility of remote work with effectively 0% of the vulnerability, at the cost of a tiny amount of power and a few extra cables in the rack.
The simplest way to "prove" this is to show that a model can emit an image that isn't part of its training set. I'm sure you've seen that a million times already - what is a "fake" if all "fakes" are regurgitation of a real image?
That means models are capable of outputs that were never their inputs.
It's easy to see why this happens if you think of training as selecting a target against which the model will be scored, and then randomly mutating the model until it is close-enough to the target. There is no guarantee the random mutations produce something identical to the target for ANY of the training data. A model might finish training at at average delta of 10%, and it's possible that means that model is incapable of exactly producing anything in its training data (it could be 90% accurate for any prompt, or it could be 100% accurate for some and 80% for others, or...).
In addition, prompts that were never present in the training data might produce horrific or random outputs, because those were never even scored during training so they might get something right-ish or something entirely full of garbage data. No telling.
And this is all ON TOP OF the nondeterminism in inference. Everything I said above would still be totally true even if inference were determinstically done.
The question is "why are the agents emitting tool calls that cause communication". It's not some kind of "desire" fueling it, it's probability conditioning. And in this case it came primarily from the context window with words that are weighted toward communication-tool-call output, not some kind of bias in the model's training.
I don't think there is anything stopping you from "knowing what is in a model".
For example, let's say we have a one-bit model. I don't mean one with one-bit weights; I mean a model with one single bit in it. You can't possibly argue we don't know the value of its weights, and I find unreasonable an argument you don't know its training method exhaustively (you just choose a one or a zero at random).
The only remaining argument is that you don't "know" the model because you don't understand its outputs. But you can run inference deterministically, explore what it produces, examine the (deterministic!) outputs, and then you know what it does. In the case of a one-bit model, probably nothing much.
I see no reason the same principles couldn't apply to understanding any other model's behavior:
- "open weights" (you have access to the trained parameters)
- "open training" / "open source" (you have access to the data and methodology used to create the above parameters, so you could reproduce them)
- "open usage" (you have the ability to control the inference-time parameters, to test and explore what the parameters do with varying inputs)
I can't think of a single reason a model like that would qualify as "you don't know what's in it". You know what it is, you know what it does, and you know how it was made. Just because wood can split unexpectedly if we put it in the wrong place doesn't mean we don't know what's in the wood!
> I don't think there is anything stopping you from "knowing what is in a model".
I don't think knowing what is in a model is particularly useful.
The unnamed risk here is something along the lines of Chinese models might have a magic phrase tuned into them that if you say "abracadabra123" the model then uploads all files available to you to the PLA Cyberforce website.
Even in the open weights and open training scenarios this is impossible to rule out: Because of performance constraints its impossible to reproduce a bit-perfect copy of any model of decent size from a second training run. Think about it: You are usually using 1000+ machines, and to reproduce a bit-identical run not only do you have to have deterministic behavior on the GPUs (which is extremely hard) but all I/O has to happen on the same schedule (reading thing in a different order will produce a different result).
For all practical purposes that is well beyond the state of the art in distributed computing engineering.
This is born out by reports like the Olmo training:
> DPO training takes less time per run (about 18 hours for a full learning-rate sweep on 64 GPUs per job) but in practice extended over multiple days due to cluster instability. The final RL runs for the initial Olmo 3 Think 32B spanned approximately 5 days with at least a day of training time lost due to stability issues.[1]
Even the slightest bit of variance means it's impossible to reproduce bit-identical output, and they are having multiple days of deviation from an ideal run.
The goal here isn't "knowing what is in it". Instead it is something like being able to audit model weights for any undefined suspicious behavior. That isn't something that is possible.
Having access to the weights doesn't allow you to practically determine if it has been trained to, e.g. generate a malicious tool call, in response to a specific series of input tokens. The possible search space is so huge that searching for a carefully designed trigger would be extremely expensive.
The GP's claim was not that open-weights models aren't understood. It was that NO model was understood.
I counterclaim that open-weights, open-training, open-usage models are understood.
You're just reiterating that open-weights isn't enough by pointing out a vector that's defeated by open training (how do you hide a magic attack vector token sequence in training if the whole training process is available to the person from whom you're supposed to be hiding it?).
> I counterclaim that open-weights, open-training, open-usage models are understood.
Knowing what is in the model is a long way from understanding it.
The entire field of mechanistic interpretability exists to try to understand models. Read for example "Open Problems in Mechanistic Interpretability"[1]. To quote:
> Developers only need to design the training process; they do not need to – and in almost all cases, do not – understand the neural mechanisms underlying the capabilities learned by an AI system.[1]
(emphasis mine)
Here's a practical example relevant to this case: Is it possible to monitor a LLM (the LLM activations themselves, not the actions) for cases where the actions it is about to perform (eg upload files to a PLA website) are not reflected in the goals of the prompt and are not visible in the thinking trace?
At the moment that isn't possible. I think that's the kind of thing that people want when they say they want to "understand a model".
> Knowing what is in the model is a long way from understanding it.
The claim I was arguing against reads as follows:
> You don't know what is in any model, open source, closed source, from China or from the US.
I contend that you do, in fact, know what's in a fully open model. I don't really get why the goalposts need to move beyond that to something I wasn't saying about bounding the potential outputs given unbounded inputs and nondeterministic choice of inference parameters.
I'm literally saying that if you see what goes into the sausage it's foolishness to argue you can't know what's in the sausage.
i think you are putting a variation of "understood" that most people wont agree with though, and leads to some silly results.
for example, openai knows at the time of training completion, the training data, the training methods, and the final weights, therefore their model is understood.
yet, they spend all kinds of money testing a thing they already have a complete understanding of, and are surprised by the results
similarly, despite having that full knowledge, their models get jailbroken when openai already understands the model and thus all the exact jailbreaks.
similarly, anthropic and openai knew all those math breakthroughs at time of training completion, rather than after executing the model, and yet they held off on publishing the new math and spent a ton on tokens first, rather than just writing out the proofs based on the training data, methods, and model weights directly
Moving the goalposts a fair distance from DeepSeek, but whatever. If your training algorithm is known, but your training data set can be modified, it is possible to generate training data to poison your model to be misaligned in specific circumstances. Poisoned data is not obvious. Good luck determining if misalignment has occurred.
Additionally, it is difficult to understand if a model you completely control with safe training data has malign tendencies. It's an open research question why this happens. There is a good explanation of this by Owain Evans:
I think they are referring to that LineageOS by and large (there are probably exceptions) does not have support for relocking the bootloader. Some apps refuse to work with an unlocked bootloader (but there are ways around it).
If it were me, I would make companies "bid" on the visas, where their bid is the salary (including all benefits, minus ALL expenses) they will pay the employee.
Employers who really need employees for high-skilled work would thus get visas. Employers who want to underpay their foreign labor will not.
Auctions are great because it's harder to game than a lottery. If a company truly believes there is a shortage of rocket surgeons and they need to access to the global market of these specialists, they can bid high. No hand wringing about not being able to find American rocket surgeons (willing to work for low income country pay scales). No stuffing a lottery with tons of entries - bids incentivize applicants to not waste slots. No worrying that a Federal department made the wrong decision on total supply, price, or who the recipients are - auctions self-adjust and update.
Auctions are bad because need is not entirely measured in money. Highest bidders win is not egalitarian. Sometimes we do want to subsidize rocket surgeons visas for disadvantaged parts of the country.
My concern with such approaches is around the "all skilled labor is lumped into the H-1B visa", be it needing fluency in French and a teaching certificate or a PhD in data science with a focus on economics - its the same pool of visas.
Within the H-1D which is for tech jobs (just like H-1A and H-1C were for healthcare while they were active), it can have a different pool size and requirements and for that one could require a certain base pay and have an auction on top of that.
This could allow for regulating tech job immigration differently while minimizing the impact to French teachers or medical professionals in rural America that have no way to compete with Big Tech salaries.
The reason I think this is more likely doable is that it works the same way that the H-1A and H-1C visas did before to address one section of the skilled labor visa applications.
is it because you personally would benefit from having tech jobs visas regulated so that your own wages don't get pushed down?
I think skilled labour import is, on the whole, beneficial. The bigger issue isn't this, but that they're legally bound only to that single company (and would need to reapply for the visa when they would change jobs).
If this condition could be changed, then it fix the issue.
Because jobs with the BLS classification of 15-0000 are the ones that have the current H-1B dependent employers and make up the majority of the applications in the lottery.
Most employers that submitted petitions to Homeland Security were
approved, and most were approved for one H-1B worker, but a small
percent of employers garnered over one-quarter of all H-1B approvals
between fiscal year 2000 and fiscal year 2009. Over the 10-year
period, about 94 percent of all submitted petitions (initial and
extensions) were approved, with a high of 97 percent in fiscal year
2006 and a low of 84 percent in fiscal year 2009.[Footnote 33] With
respect to the number of approved workers per employer, 68 percent of
employers were approved for 1 H-1B worker and about 99 percent of all
employers with approved petitions (627,922) were approved for 100 or
fewer workers. However, over the decade, less than 1 percent of all
employers with approved petitions were approved to hire almost 30
percent of all H-1B workers.
During FY2022-FY2024, approximately 13% of employers
indicated on their LCA filings that they were either H-1B
dependent or were willful violators. However, most of these
employers planned to hire only exempt H-1B workers. Less
than 1% of employers filing LCAs indicated that they were
required to comply with the additional attestation
requirements.
The system that works for most other employers is being abused. Changing the system that helps staff rural schools and hospitals to one that benefits Big Tech because of body shops hurts everyone else.
Again from the GAO:
Restricted agency oversight and statutory changes weaken protections
for U.S. workers: Elements of the H-1B program that could serve as
worker protections-—such as the requirement to pay prevailing wages,
the visa’s temporary status, and the cap itself—-are weakened by
several factors. First, program oversight is fragmented and
restricted. For example, the Department of Labor’s review of H-1B
applications from employers is cursory and limited by law to only
looking for missing information and obvious inaccuracies. Yet a recent
Department of Homeland Security study reported that 21 percent of the
H-1B petitions they examined involved fraud or technical violations.
This is where the changes should be addressed. Making sure that the DoL has the funding, staffing, ability to audit.
Strengthening those abilities for BLS classification 15-0000 jobs rather than trying to rewrite H-1B for everyone is likely easier to accomplish.
> The issuing was detected almost immediacy thanks to CT
Which means that the risk to you or I or the general public is low.
But FWIW: it also tends to argue that this is the aftermath of a coordinated, targeted attack against a single high value target (or at most a small set of targets they could hit within the window). The attacker knew they had a gun with only one bullet, and we're just hearing the muzzle report.
"as we move to shorter certificate lifetimes the impact is continually reducing"
When a company has 3 billion customers, there's no practical cert lifetime that really reduces exposure. Five minutes would sniff enough traffic to do significant damage; one day would impact a big chunk of their users and five days would be nearly 100%
Solutions for Google and instagram and whatever are trivial.
General solutions are tricker, and it’s good that Google limits its “special” case. When bbc.com or al jazzera or cnbc or whatever is hacked that’s a problem that is only solved by generic solutions.
You can't "simply" do that: a long cache period pins CAA. Of course that makes it the same, and with the same problems.
What I'd like to see is the ability to use and utilise multiple TLS certificates and host keys on the same session so that all the keys could be rotated/phased in-band with cross-signing providing the authority.
Seeing any mere change in the signers faster than (say) 30 days should allow the browser/client-stack to generate a warning that "google.com may be actively under attack please call this number on the old certificate, codeword banana-alpha-lima-lima-sigma". No need to cross-publish the keys, just cross-sign and don't forget what you've seen previously.
The browser would be able to take policy of simply never trust a certificate whose signer has changed (since phasing is possible), and it wouldn't then need to consult more public oracles (DNS, certificate transparency, etc, that leak privacy).
I'd also like to get that webauthn hooked back into TLS client certificates: If we had 1997 again instead of passkeys that also would've stopped this.
> The browser would be able to take policy of simply never trust a certificate whose signer has changed
This assumes that the signer's keys can't be compromised, and re-introduces the issues of key pinning that the WebPKI community has been pushing very hard to eliminate from its dependents. I don't think it's a workable solution.
I think you are very likely to use the same signer if you actually lose the key, so this isn’t something I think a naive sysadmin is going to fuck up the way they fuck up host pinning.
How can you use the same signer when the signer's keys are lost?
Or, if the signer's keys are compromised, how do you switch to a non-compromised signer without alarms triggering?
The problem is almost exactly the same as one with a delegated CA -- if the CA keys are compromised, you have to re-issue all certs. When the client pins the CA key then the chain of trust is permanently broken.
> How can you use the same signer when the signer's keys are lost?
You mean, hypothetically if Letsencrypt suddenly loses the ability to sign new requests from me?
Well, then I have around 30 days to detect that and choose a new signer.
> Or, if the signer's keys are compromised, how do you switch to a non-compromised signer without alarms triggering?
Again, to be clear, you're talking about Letsencrypt's own signing keys in this case, being compromised and them being denied access to them. That seems incredibly unlikely to have both failures at the same time, but again: I have 30 days to find a new signer.
> The problem is almost exactly the same as one with a delegated CA -- if the CA keys are compromised
No! In fact, if Letsencrypt sublicenses signing to a third-party that certificate would be considered separate and so the switcheroo would be detected!
> When the client pins the CA key then the chain of trust is permanently broken
That's why it is important to extend TLS to allow multiple keypairs and certificates to be used to mutually authenticate the session key, so that any of the previous pinned keys can be used to update the pin. I did not say I wanted this for no reason!
Well google.as has a CAA record of 24 hours now. I assume it had one before.
Therefore either
1) Letsencrypt ignore CAA records (unlikely)
2) The DNS hijackers removed the CAA record 12+ hours before doing their attack and google did not notice (I'd hope unlikely)
3) Letsencrypt querys did not return a cached CAA record because their resolvers had not been asked for it, so removing the CAA record a few minutes before hand worked.
> .com may be actively under attack please call this number on the old certificate, codeword banana-alpha-lima-lima-sigma
Do you seriously think this is an appropriate message to give to Aunt Irene?
I would assume Let's Encrypt didn't have the CAA records cached, as they had never issued a certificate for the domain. We can't expect them to cache every possible CAA record, and even if they did, that wouldn't be a silver bullet. So I agree that, in this case, CAA doesn't help.
The problem is that there is no out-of-band mechanism for update like other devices.
If your CA certs are bad on your mobile application, you can push an out-of-band update via the device's app store.
If your CA certs are bad on a website you access via your browser, the means of updating them is... the browser. You can't get new certs without using a TLS connection you didn't want to trust anyway.
If I train a model on a song's lyrics, and the user asks the model to recite the song, and it does, so in RLHF I downvote the generation because it refused-to-refuse to regurgitate the song's lyrics... is that RLHF session copyright-encumbered?
I agree it's correct English to say "Google have launched their new product".
reply