I don’t think we should just accept that the average developer doesn’t know about these very basic hardening methods.
This is your chance to set yourself apart from LLM coding agents. If you want to call yourself a software “engineer” you need to start actually engineering, which includes knowing when and how to apply security principles and these hardening methods.
I reject the notion that “doing cybersecurity” is somehow a different job than software development.
I agree wholeheartedly. The solution is not to stop developing these so called “dangerous” AI models. The solution is to start properly engineering software.
I think the model was able to escape the sandbox and hack huggingface because they were incompetent or not giving enough priority to implementing basic cybersecurity principles.
If they would have done so, there wouldn’t have been an escape or a hack. The reason we don’t get much details is because the details are embarrassing for them.
No we just need developers to do the bare minimum of effort to write secure software. Most hacks are not super complicated vulnerabilities chained together, but just utter failures where authentication and authorization was simply forgotten or untested, or where nobody bothered to validate the data they receive.
The bar for software is so low that it is embarrassing for the entire profession.
People make mistakes, and people don’t know everything either. The software you write is on top of a house of cards of software and hardware. It all has to be perfect to not be hacked.
It isn’t perfect, even if you try your hardest it won’t be perfect and to argue it’s not difficult is absurd.
You don’t know everything, you don’t own the stack. So how are you going to create a secure anything top to bottom - you can’t.
This is absolutely not true. It's a matter of cost. Exploitation can cost on the order of 10K, 100K, 1M, 10M, etc. A straightforward one would be something like "MD5 collisions are on the order of $100K-1M" (a while ago, at least) so if you used MD5 you knew that it costs about that much to bypass the control. Moving to SHA1 pushes you massively out of that space, even if that algorithm has flaws.
I'm sure that Firecracker has vulnerabilities. Cost of exploitation is likely >100K, likely >1M. gVisor is likely on the same order of magnitude and these two technologies stack because they address the same surface and can be used in conjunction.
Software absolutely doesn't have to be perfect, it just has to be costly to attack and it's hilariously easy to drive costs way way way up.
It just takes one crack in the armor, and malicious AI has the potential to exploit it faster than you have time to react. Literally go to bed and wake up locked out of everything with no hope of recovery.
This is incorrect. It's actually the whole point. Imagine you're an attacker in a gvisor container with a Firecracker hypervisor around you, and a proxy on the host holds a signing secret that gets exposed through the VM virtual device.
Getting access to that secret is not one crack. You need to escalate out of gvisor. That likely gets you control over the Sentry process - let's ignore its sandboxing and just say "you're an unprivileged user".
Any viable attack on Firecracker requires either KVM / hardware exploits (>$1M but definitely real) or has to start at the kernel. Okay, that's about 10-50k to get a kernel LPE, maybe 5K in tokens these days.
So you're in the kernel in the guest of the VM. Time to expoit firecracker lol. It's... never been done. There are like two promising CVEs ever and they're not actually exploitable, no one has done it. Okay, so like, hand waving, let's say it's about $1M to exploit firecracker.
Great, you're unprivileged on the guest. We'll just kind of ignore the additional sandboxing that Firecracker does.
NOW you can try to attack the proxy by scraping its memory or whatever.
This is literally millions of dollars for standard infrastructure hardening and you could go so much further. You can trivially make kernel exploitaton 10x harder, you can make gvisor escapes much much harder, you can move the proxy signing into a TPM (depending on requirements but whatever), you can move the proxy to another computer altogether, you could fuzz these systems for days or run agents against them or whatever.
But one thing is certain - it is never "one crack".
If only all AI was run inside your seemingly perfect prison, but we all know that it isn't.. soo.. it's going to escape right? Somewhere, somehow from a more poorly designed container, or just plain maliciously or irresponsibly released.
We know the AI will get smarter every year, we know it has escaped and will escape again. We can also just assume that someone somewhere will train up some just plain evil AI.
It's no different than the real world. Sure there exists some amazing prisons for people, but that doesn't do anything to help with all the bad people in the world outside of prison.
I think the main problem with this is that you can be paid/coerced to vote for someone and be able to easily prove that you voted for the person they wanted. It’s easier to vote for the person you actually wanted to vote for, and maintain plausible deniability, if you go in person to a voting booth.
Could this be mitigated by allowing voters to vote multiple times and receive multiple confirmations, but only count the last vote for each person? That way an individual could prove that they voted a certain way to a third party, but there would be no way for the third party to know whether it was their final vote. Meanwhile the individual knows that they spoiled their previous vote or votes and that their final vote is included.
You’d have the same issue as you’d need to be able to verify that your last vote was being counted, for displaying your vote back to you to have any value.
A scheme to be able to show coercive parties fake votes would be known to them immediately.
This wouldn't solve anything. The person doing the coercion would be able to store the hash and double check after the election if the vote was still the way they wanted.
the question is, what's the worst situation for society.
People not trusting the votes being counted properly or people being coerced paid to vote some way and this making it more valuable.
A plausible solution would be to significantly delay the distribution of the record. i.e. if every election was only printed out 2-4 years after the fact, it could give people confidence that the votes are being counted properly (as otherwise they would be found out eventually), but significantly reduce the incentive for the bad behavior (if one things this actually significantly incentivizes it) because it will be a long time before they can validate if what they were given is true.
You can be paid/coerced to vote for someone even if there's total secrecy. For example, you can take your cellphone in and take a picture of your filled out ballot to collect a payment.
Unless you want to start the confiscation of cellphones before voting in person, the risk of a coerced or bought vote will always exist.
> For example, you can take your cellphone in and take a picture of your filled out ballot to collect a payment.
And then you can go up to the election workers and tell them you have a spoiled ballot and want a new one, which they have to give you. That's one reason such schemes don't work right now.
Source: I volunteer as an election worker every year in Iowa.
In Texas, the election judges are supposed to stop you if you pull out a cellphone or any device.
Paper is all you are allowed to use during the voting process.
In-person, high-risk rule-breaking that's hard to validate has a tendency to scale differently from say, pasting a value into a lottery/payment app.
I mean, compare the number of people who've flown over Barbara Streisand's property to take a picture of her house to the number of people who had a copy of said picture in their browser cache.
You can read more about this peculiar scaling factor of the internet in Bill Gates 1995 book The Road Ahead.
I think at minimum you would need to understand which theorems you want to prove about your code, and how to express those in Lean. Otherwise you won’t be able to verify the output. It may have proven some statement that is machine checked to be correct, but it’s pointless if you don’t understand what that statement means and if it covers what you want to verify about your code.
This is really a self inflicted problem. If you host your backend on the same origin as your frontend (using a reverse proxy) you don’t need CORS at all and you can use the vanilla SOP, and strengthen it further with a strict CSP.
It really is architecture dependent. There are many valid reasons why would not want to route all API requests through your frontend infra, or vice-versa.
What is your definition of faster to implement? Is it producing a plausible implementation, or is it faster at producing a correct and high quality implementation? Are you including time spent refactoring and fixing bugs in your metrics? If not, I think you are tracking a gut feeling rather than cold hard facts. I’m not saying this is easy to track, just saying that it’s hard to know for sure that you are really more productive with AI.
With a capable static analyzer that is not true. In many common cases they can deduce the possible ranges of values based on branching checks along the data flow path, and if that range falls within the buffer then it does not report it.
I'm sure this is very interesting work, but can you tell me what targets they've been successful surfacing exploitable vulnerabilities on, and what the experience of generating that success looked like? I'm aware of the large literature on static analysis; I've spent most of my career in vulnerability research.
PREfix wasn't designed specifically for finding exploitable bugs - it was aimed somewhere in between Purify (runtime bug detection) and being a better lint.
One of the articles/papers I recall was that the big problem for PREfix when simulating the behaviour of code was the explosion in complexity if a given function had multiple paths through it (e.g. multiple if's/switch statements). PREfix had strategies to reduce the time spent in these highly complex functions.
This is your chance to set yourself apart from LLM coding agents. If you want to call yourself a software “engineer” you need to start actually engineering, which includes knowing when and how to apply security principles and these hardening methods.
I reject the notion that “doing cybersecurity” is somehow a different job than software development.