Most chat support people were contractors hired from third party companies that were given dossiers about their products that were often quite out of date because poor management has always been a thing.
Last time Coinbase had a breach I wrote in to support chat to see what I needed to do. They said there was no breach. I sent them a link to their own blogpost about it. They responded with "wow this is the fist time I'm hearing about this."
> At minimum, a half-way decent support person would ask a few people internally or search Slack before answering.
Of course not. The extremely vast majority of support staff aren't connected to "internal people" and certainly don't have any access to the main company's Slack.
Most of all, those people are paid very little on very tight length-per-interaction targets. They can't spend any time at all looking for stuff outside the docs package or chatting with peeps outside the immediate costaff.
Its an organic cycle, as the company grows big the number of relevant areas grow too and inter team communication becomes way too costly/impractical. The big company becomes a group of informal small companies, each running in their own direction and at times competing with each other. Areas like customer support are not really good candidate for career growth so they get least resources and manpower.
Everything I said applies to overseas labor, too. I’ve worked at both types of companies, those that co-locate support with the product teams, and those that pay bottom dollar and don’t care at all about support.
With the latter type, there’s still almost always a line of communication to corporate. And the support staff still try to help. They are decent human beings, even if the end result kind of sucks.
The vast majority of people I know who have done phone/chat support have had a spreadsheet (or fancier tool) of common phrases and mandatory boilerplate answers, an overview of the company they're doing it for (sometimes just a couple pages in a word doc), and tone/tool training at a minimum. And generally that tool has a tree of common steps that are also generally mandatory, because they punish rather harshly for going off-script. It's not much, and it gets you the sort of support that people often think of with ticketing systems: impersonal and inaccurate, favoring the company.
But it's rarely this inaccurate. Or if it is (e.g. missing a major product launch), it's fixed in a day or two.
And as a dark pattern it adds "positive friction" for the company reducing the number of people that will have the motivation of obtaining the real people support.
Several weeks ago I had an issue not being able to login to Verizon's website, so I tried to chat with someone. The chatbot that was gatekeeping was predictably useless said it would redirect me to a human except...it kept prompting me to log in first. It was literally impossible to differentiate from if they literally had no humans online to talk to at all.
I could be the best money saver for them - and ask for a hefty premium for my services - by terminating all support. No costs, nada, full save! Genius, right?! Never gives false info, never!
Ok, ok, need to have a tickmark next to the 'support' item in the quarterlies, let it be an eternal spinning wheel presenting on clicking the 'Our award winning instant support is HERE!' button then. Its close to the real experience anyway, right?
I generally agree with all of this, but I'll add a few additional remarks. Because it's come up a bunch lately, I decided to do a bunch of code review/audit of the Fil-C codebase, and I'd say while it's got a lot of good bones, there's a long way to go to being a foundation I'd be ready to build on. I've reported a few UAF's upstream, and I've got a few PRs I'll add on, but if it only took me a day or two to find some of these big holes, I'm sure there's more lurking under the surface. I'd consider it to at this point be more of an engineering demo that this approach is feasible and tractable, but not a production ready language that I'd want to ship code in.
On the muddling the discourse, I'm not on twitter and don't engage there, so I don't have an opinion on that, but I did come across https://news.ycombinator.com/item?id=49044561 recently, and I just don't see how the author can make such bold claims while examples like the one Steve provided above are still in the language. Corrupting memory in Fil-C is still easy, type confusion is still easy, intra-object overflows are still easy. Fil-C prevents a range of classes of bugs from being exploitable, but it doesn't stop the bugs from happening.
Yeah. In that thread the Fil-C author said of typescript, go and C#:
> Those languages rely on a much larger pile of YOLO C/C++ code for their runtimes and standard libraries than Fil-C does. So Fil-C is safer than those
Given the relative immaturity of Fil-C, this seems wildly wrong to me. I’m not sure how to take his claims about his runtime seriously.
This is not a simple fact that follows, and a good example of why you seem to be catching so much criticism for overly bold claims. One could state that a smaller runtime is easier to audit, and so the investment needed to reach similar levels of safety is lower. One might even argue that after similar levels of investment, that the probability that it's safer is higher. But jumping all the way from lower number of lines => safer is a simple fact is a huge leap.
I have worked on both a major JS runtime and a .NET runtime. I have fixed hundreds of security bugs in JSC. Based on that, I have no doubt that what Fil-C does is safer. Just the absence of a JIT makes it safer in a way that I don’t think is seriously debatable. Even with JIT disabled, a JS runtime has massive attack surface due to the language relying on a large native library to do anything useful, not to mention a mind boggling amount of language implementation corner cases.
By contrast Fil-C has a small number of rules and largely obviates the need for “native” code.
First of all, it’s incredible that on a HN thread about a language that isn’t C, there are 46 mentions of Fil-C! You guys are obsessed!
I make bold claims because they hold water.
- You can at worst corrupt only the capability you’re pointing to.
- intra object overflows are almost never useful for memory corruption exploits unless they let you corrupt a pointer, and Fil-C prevents that from being useful because you cannot corrupt the capability.
- the zunsafe api is basically unused. One library uses it (OpenSSL) for good reasons. This is in contrast to widespread use of the unsafe keyword in Rust, beyond just one library for a narrow purpose.
Thanks for reporting bugs. Worth noting that they require doing things that extant C code never does. It’s good to fix those, but the true threat model of any memory safe language is not to sandbox a malicious programmer, but to protect the program of a normal programmer against a malicious user
> the zunsafe api is basically unused. One library uses it (OpenSSL) for good reasons
Wait, so there are escape hatches? But… you’ve repeatedly said, many times, that there are zero escape hatches?
And now here you’re saying not only that there only are escape hatches, but there’s a good reason to use them?
Damn. Misrepresenting `unsafe{}` whilst saying your language is better because there are no escape hatches and no need for escape hatches is like… 80% of your online personality.
When can we expect the website to be updated to remove the misleading claims?
"Fil-C has no unsafe statement and only limited FFI to unsafe code."
`zunsafe_call` is a weird thing to get hung up on as an "escape hatch", considering it's just a super limited form of FFI, intentionally designed so that it's only usable for OpenSSL's use case.
> Misrepresenting `unsafe{}`
`unsafe` lets you write Rust code that violates any reasonable definition of memory safety (including Rust's definition or my definition), and it's widely used.
> Unlike other approaches to increasing the safety of C, Fil-C achieves complete memory safety with zero escape hatches.
Except there is an escape hatch, by your own admission above?
> `zunsafe_call` is a weird thing to get hung up on as an "escape hatch"
from the docs:
> unsigned long zunsafe_call(const char* symbol_name, ...);
> Performs an unsafe call to Yolo-land.
That’s just a `unsafe{ func(…) }` escape hatch
> intentionally designed so that it's only usable for OpenSSL's use case.
Cool motive, still an escape-hatch =)
Do you have the backbone to update the fil-c website to correct the record, and let the person you retweeted here[1] know that the escape hatch row is incorrect?
I think the consensus is that this would be the wrong end to tackle the problem from. Your project should include ruff at the version it wants to use in its pyproject.toml. Bump the version when you're ready to spend the time on one project, but no need to coordinate across multiple projects. If you really want to do it, you can, but if there's one hold-out, you're not stuck behind forever because one project can't move yet.
The really silly thing is that Fil-C and Zig here have chosen a definition of memory safety that specifically excludes a bunch of overflow vulnerabilities and type confusion. At least it’s been specifically defined, but such a narrow definition makes the end result to me almost wholly uninteresting. UAF no longer turns into a RCE, but your average parsing packet code is still just as likely to be a buggy mess.
I’m having a hard time understanding how these two things can be true at once. Just because the capability exists that somewhere in my program a valid pointer to a piece of memory might exist (what I understand accessible to mean), doesn’t mean that a particular write to that memory location under a data race that tears a pointer is valid. The data race may make a pointer that would never algorithmically appear in the program in the absence of races, and thus make writes that invalidate invariants that should have been preserved. It seems to me this still provides a way to corrupt memory in a difficult to debug way just like C. Am I missing something?
I've spent some more time over the last day looking at Fil-C more closely, and come to the conclusion that this:
> The thing that makes races hard to debug in C or C++ is memory corruption; that doesn’t happen in Fil-C
is just blatantly false. It's almost as easy to corrupt memory in Fil-C as it is in C. Type confusion is still allowed. You can still do out of bounds buffer reads/writes as long as they fall within an allocation, so any intra-object stuff, as you might get in common packet or binary file parsing, is still about as unsafe as it was before.
> Given the absence of guardrails there was nothing to prevent the model from attempting to break out of that sandbox, break into Hugging Face, and read the answers from there instead.
I've said this many times before and I'll continue to shout it, but using the term "guardrails" to refer to anything that's either (a) in-context, or (b) a probabilistic classifier (including using other LLMs), is an irresponsible abuse of terminology that we as an industry need to put a stop to. Guardrails are the actual systems we build in place around these things that deterministically bound the permissions, not prompt engineering, not RLHF, not external LLM-based classifiers. I believe those types of "guardrails" are a result of a combination of fundamental laziness: they're faster to do than doing things correctly, and a result of too many folks involve being AGI-pilled, thinking we're just one more model away from this all being so smart that it just understands what they mean when they give an LLM some fuzzy language rules to follow.
There can and should have been additional real guardrails put in place here. Zero-day or not, breaking into what should have been an offline, frozen package cache that also does not have internet access should have been insufficient. Network level protections should have identified the traffic to the internet originating from this network as an anomaly long before there was time to exploit an outside company. These are not new and unknown problems, the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part, especially given how much they like beating the drum on how dangerous these technologies are. Shame on them, and honestly, shame on Simon in this article for accepting the broken terminology that they continue to rattle off and calling them out on their half-assed and demonstratively inadequate approach to security.
What we call "guardrails" in an AI agent, we would refer to as "honor system" in human actors.
Or, in a more direct sense, the AI should be set up in an environment such that no matter how hard it may try to call $PART_OF_EXPLOIT_CHAIN, the environment just isn't capable of it (ideal) or doesn't permit it to do it.
I like "honor system" as a term. I've been looking for the right term to replace the irresponsible usage of guardrails with, and best I've had so far is the pinky promise protocol.
It's worse than an honor system, because humans are constrained by social forces to some extent, whereas we don't know what AI is or how it will behave
I agree with every word of this except "irresponsible". We don't have enough information to say anything for certain. But based on their incentives and track record of similar behavior, the burden of proof lies with OpenAI to prove they didn't prompt the thing to achieve this exact outcome. The most likely scenario is that they were purposefully executing their responsibility to their shareholders to produce their own Mythos moment.
Anthropic's Mythos moment earned them a two week period where they had the best available model and couldn't sell access to it... and by the time the US government allowed them to sell it again OpenAI had released GPT-5.6 and Fable was no longer undeniably the best model.
These things don't have a long shelf life. Losing two weeks of on-sale time for your best model is bad for business.
A fundamental misalignment in US capitalism is putting the business and revenue as the #1 priority far above all other aspects in society. They have good margins and the Chinese are doing the same on the cheap by comparison. US Big AI can afford to bear more of the burden.
Nonsense it was irresponsible. These are all steps threat researchers use to isolate and test real malware whose behaviour is essentially the same in this case for AI
I've seen plenty of fucked-up guardrails that vehicles have passed through. We know that with enough momentum they will be penetrated. It seems like exactly the correct terminology to me.
The term guardrails in both the original physical sense and in cyber security implies a weak safety control - they can help prevent accidents, but they are not strong security boundaries.
Nevertheless we collectively don't expect that road and car safety infrastructure should be limited to the front and back fenders on the cars themselves. And neither we even mention those as any notable safety equipment usually, despite each car carrying them. The main safety infrastructure in any industry is something external to the dangerous object. Car fender doesn't 100% protect a pedestrian from being hit and neither does a metal rail on the side of the road. But the effectiveness of both are "slightly" different.
i've always been under the assumption that "AI Safety" is baked into the training of the models and not a parameter that can be turned up or down. So if someone breaks into Anthropic one night and makes a full copy of Mythos or whatever then that model they copied is fully capable and not lobotomized? That raises questions because, if you believe all the PR, that's equivalent to breaking into a research university and stealing an entire bio/chem weapons research department.
edit: if the above is the case then we should just assume it's already happened because of the value to both goodguys(tm) and badguys(tm).
There is some aspect of baking the rules into the model, that's what I refer to as RLHF above (which I use here as more a catch-all term for a variety of post-training activities), but there are also external to the model classifiers that may run on the prompt input or on proposed tool calls to limit the way in which the model is used.
The irony is in this case the in-context and classifier "guardrails" would have almost certainly stopped the attack while their attempts at your definition of guardrails (the sandboxing) failed. In general, people keep trying to make secure systems and they fail with surprising regularity. Saying "they should have had better security" every time someone gets hacked is perhaps true, but it's not going to stop hacks from happening. And it's not a sufficient strategy on its own against future LLMs. "The Bitter Lesson" probably applies here.
Issue is when saying what can’t be done, there’s no funding, in industry or modern academia. I did say the same thing long time ago, not that it matters, the industry goes on its own way. Rightfully so in this case as it turned out, since things nowadays are not the same as a decade ago.
Fair on the terminology angle, that said in this case, they had proper guardrails no? They were running in a sandbox, but it was able to find an exploit out of the guardrails.
Clearly not. Good security is about defense in depth. The service that was exploited was meant to be an offline, frozen package cache, it shouldn't have been anywhere near a network boundary to even allow this through. A single exploit in one service should not have been able to bypass all network filtering, monitoring, and alerting.
Exactly, that's why this story is so crazy no? It's saying the model was so good at hacking, it managed to bypass all of that, and also found a way to get into Hugging Face.
I can understand skepticism, did they have poor security measures, or is the model that good? At least the way they portray it, they had top-notch security measures in place, and yet the model managed to escape confinement and also hack Hugging Face.
Why was the computer physically connected to the internet though, sure, they could have gone that far as disconnecting it physically, but my guess is that's a cloud box in a data-center they provisioned, running on an entire rack cluster of GPUs.
I was not aware that the term "guardrails" has a universally agreed upon definition. I've certainly seen probabilistic classifiers referred to as guardrails many times by many different people.
I usually don't use the term much myself because I don't think it's clear and I ambiguous, but I stumbled and let it sneak into this piece. I think I was influenced by the Hugging Face post I quoted.
I expect OpenAI would agree with you that "the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part". They have clearly invested a lot in those systems for their production models, but in this case they had deliberately turned a bunch of them off for a research project.
I think their biggest mistake here was not VERY closely monitoring their research box here. They should have noticed and shut it down the moment it broke through the package proxy.
> I expect OpenAI would agree with you that "the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part". They have clearly invested a lot in those systems for their production models, but in this case they had deliberately turned a bunch of them off for a research project.
I'm not sure I can reach the same conclusion. There's no reason that stronger network protections would need to be turned off for a research model that's not supposed to have internet access. There's no reason that it they couldn't continue to run in whatever hypervisor or VM that any other production systems could use. The only thing that should be turned off for this type of work are the non-deterministic classifiers that they use for auto-approval mode in their harness, and I would wager that's the only one they did turn off.
> There's no reason that stronger network protections would need to be turned off for a research model that's not supposed to have internet access.
They didn't turn off the network protections - it looks like they used the same system as other sandboxed agents, which is to block all network access except for an HTTP proxy which only allows read traffic to PyPI and a few other package installation targets.
I'm hoping a more detailed retrospective clarifies if the vulnerability the model found in that proxy affected their other production systems as well. It sounds like a data exfiltration mechanism, and I have a keen interest in those.
My current hunch is their biggest error was not using the same network monitoring on their research machines that they use in production. Their network monitoring really should have spotted what was happening as soon as the model broke out.
(It's also possible they were running the eval on a developer laptop somewhere!)
Right, but this is my point in saying I can't reach the same conclusion that they've invested a lot into these systems for production models. Either they had better network protections, and turned them off in their "sandboxed testing environment" (their words), or they don't have more comprehensive network protections at all and rely on this one, extremely thin layer even in production.
There's no such thing as a "deterministic" guardrail. Such a tool is as likely to be exploitable as an LLM is to break out of its probabilistic conditioning. Have you never driven past a destroyed guardrail on a highway? These systems are always best effort.
Add to that list respecting the TOS of the user's ISP so that the user does not get banned, and providing some sort of remuneration if illicit activity through the proxy causes problems for the primary user, and then _maybe_ you could call this all not shady as fuck.
If only ISPs would actually crack down on (usually unwitting) users whose systems are participating in a malicious botnet or otherwise have their networks compromised. They could offer temporary disconnection + anti-malware support to get the user cleaned up, if they actually gave a shit.
No residential last mile broadband ISP at the scale of hundreds of thousands or millions of customers is presently willing to spend the salary/benefits/fully loaded employee cost on the massive teams of (somewhat technically clued in, not low wage) people it would take to effectively implement this.
reply