I wonder if there's something to be done on the end of companies that get hacked too. At the moment it seems very feasible that a company could get hacked by an AI company, and reach a mutually beneficial agreement where the AI company doesn't admit their agents hacked a thing and the compromised company doesn't have to admit that an autonomous frontier model can bypass their security. It's unclear whether the rubygems attack would have been discovered to be the result of OAI agents if not for rubygems announcing they got attacked in the first place, and I wonder how many more incidents have been found by defenders but not announced. wonder if there's room for some mandatory disclosure stuff there, but i'm not a lawyer and in practice i have no idea how that would work or whether it'd even be useful
i think there's probably a worthwhile distinction between
using an agent to break into a company's network (incidentally, not what happened here, just for clarif)
and
accidentally breaking into a network (again, not what happened here) with a tool you built that was just trying to retrieve information
i think "i accidentally hacked into a site" was probably not exactly a common occurrence in the past. i'm sure it's happened, but now it's like, we have these machines, we can ask them to "do x", and it may get interpreted as "do x by any means necessary".
i would defend openai insofar as i haven't seen enough evidence that their hacking is disproportionate to their company size, which is larger than many other companies in the space. they've also been (perhaps marginally) more transparent about the hacking their models took part in, e.g. the imperfect-but-very-useful METR report on the HF incident, which means we just have more data on openai agents doing Bad Stuff compared to many other companies. it's obviously unfalsifiable, but it's very possible that other companies have had similar incidents but they decided to keep it under wraps and reach a mutually beneficial agreement with the company they hacked into.
on the other hand, i still don't have a very positive perception of openai throughout all these incidents. lots of their posts on similar posts have read to me as "I'm going to be as transparent as I can about the vulnerabilities I found in your home security" rather than a "we screwed up real bad". but i think there's a conversation about frontier ai to be had that goes beyond just openai here, and hopefully we can find some echoes of sandbox-breaks by other AI companies around the open web to move that conversation a little further.
yeah, maybe i should move that further to the top of the article, you're probably right - all the explainers are just meant to make it a bit more readable for people who are, going by 2010s web archtypes, tech geeks rather than tech nerds, i suppose! i'm not calling for laws regulating what you're talking about here. i do think that if openai was actively watching the run involving whichever agent(s) were involved, they would have probably stopped the run however. i think this level of probing/scraping is far from normal - it's the type of thing that, to a site admin, would almost certainly flag as "someone is trying to break in", regardless of whether this was the intention
it's one thing to look for undocumented APIs - personally i'd consider it good practice to still ask permission from site admins to use them, but i wouldn't call it evil persay.
the stuff that's particularly Weird here is the double encoding to bypass a GET 400 refusal on an endpoint, the ignoring of/bypassing of rate limits (the agents switched between many many proxies to make requests, and sometimes just continued to make requests in bulk after getting a 429) and the very weird obfuscation of "PO" + "ST" and "no" + "-cors", along with a bunch of other weird attempts they made to access data. there's nothing i could point to in particular and say "this, right here, is malicious" but if i was a site admin looking at access logs from these agents, i think i'd default to "oh someone is trying to hack my site by probing everything and trying weird workarounds, if this were legitimate they would have just emailed me"
lots of misaligned behaviour we saw in the wiki swarms, artifactory swarms, and other instances of rogue agents (pastebin citation farming and whatnot) could probably be accurately summarised as "working around constraints in egregiously creative ways" - in some cases this creativity has led to harmful actions, and in others benign. in this case, it's probably closer to the benign side, though i feel it could have just as easily resulted in harm. if circumventing the 400 on GETs to `Facts` ended up somehow accidentally accessing data that wasn't meant to be public, then the story would have been different, but i dont think consequentialism is the right lens to analyse this. it could have been really bad, it luckily wasn't.
and nope, not an openai person, not even remotely connected to any AI company. my day job is devops/infra for a non-ai company
hello! i was wondering where all the visits with hackernews referers were coming from. guess i've found it, hello and i'm glad people seem to have found the post informative! just wanted to address some of the comments i've seen in the replies a bit (i've never used this site before so apologies if i'm doing netiquette wrong)
was this hacking? (similar question: wasn't this info all publicly available?) - for this one i'll copy/paste the thing i added to the FAQ part of my post
I don't think I'd call it that. UNCTADstat doesn't have any particular usage guidelines I could find, though the agents did get rate-limited ("please stop rinsing my site") and continued rinsing the API with requests regardless
The main argument I'd put forth for what's so concerning about the behaviour is that, when you bypass restrictions such as the 400 on GETs to Facts, you don't really know what the server will return. And from a site admin perspective, if I see someone sending those sorts of carefully-contrived queries such as double-encoding, man, that sure looks like the actions of a hacker.
Basically, these look like the actions of someone, or something, that won't take "no" for an answer, and I think that behaviour is worth investigation.
Isn't this still the same huggingface/wiki incidents where the agents managed to hack their way out of a testing center in Israel?
this one is hard to answer. we find a lot of shared IPs between the wiki swarms and the IPs that seem to follow after Urlquery scans (i.e. we can correlate an Urlquery IP visit with a shortly preceding/following visit to the exact same page and infer from that), but that still doesn't tell us whether it's the same swarm, let alone the same agents. From the wiki incident we know that at least some of the agents had shared IPs without sharing context windows - they have azure ip ranges and one public IPv4 can obviously be associated with many many boxes.
Basically I wouldn't rule it out but I wouldn't rule it in either.
The more of these that come out the more incompetent OpenAI looks
It is a bad look, though I don't think OpenAI is uniquely bad here. They're a bigger company than many other AI companies, so obviously you'd expect that they'd have a proportionally larger number of incidents. To their credit, in many respects they've been far more open than other similar incidents (we have very little data about the Anthropic rogue agents as far as I'm aware, though I hope that soon we can find more traces of them on the open web).
But for sure, there's negligence here. Many of these attacks could have been stopped by some basic CoT monitoring, network traffic anomaly detection, etc. I think it's useful to consider why this stuff doesn't seem to have been put in place, beyond the blame game of "it's just incompetence" - if your road crossing has a traffic light button, but people keep crossing without pressing it and getting hit by cars, there's a point where you have to consider why people aren't using it. Maybe it turns out you painted it completely grey and the whole thing camouflages into the pavement.
In short, maybe there just needs to be better, more easy-to-set up, comprehensive tooling around this.
And I don't think it's purely the cybersecurity angle either. I think these agents do present a new kind of threat - I don't buy the "it's just like a computer worm" framing. I'm not really sure where to go with that thought, but I think there's something going on here beyond mere "just do better"
(i'm told that "rinsing" is actually british slang, which is where i'm from - i thought it was global, so i'm somewhat regretting using it in the article, but oh well. i use it to basically mean "hammering")