Bold of you to assume that AI-written and -reviewed code doesn't introduce new vulnerabilities. I think after we're done with this stretch of finding human-made vulnerabilities, we'll have a long road ahead to clean up the mess that's introduced by AI.
> Bold of you to assume that AI-written and -reviewed code doesn't introduce new vulnerabilities.
If AI can find a vulnerability in a given piece of code, it can also do so during testing, not only when used by a malicious actor. This means that the whole thing becomes an issue only if someone is writing a safety-critical program and not doing proper LLM safety testing.
Everyone should use LLMs to test the safety of their code today.
"Should" is doing a lot of heavy lifting there. Most shops are a long way off from secure coding, and from my experience in the last 10 years most devs lack basic awareness of security. I still see people being surprised by OWASP Top 10 or simple things like trust boundaries.
There's no guarantee AI will find existing vulnerabilities proactively during coding, or that it won't introduce new vulnerabilities in the process of shipping features. You're right that shops _should_ be critical and test code, but I think due to the lack of awareness people will end up implicitly trusting any first pass written by AI and not bother to double check it.
It's been shown to be bad at secure coding _unless_ you specifically prompt models to watch out for common vulnerabilities, at which point they're actually quite good at containing the issues, though the studies here are somewhat older (eg https://arxiv.org/abs/2211.03622).
The issue then is that most people aren't actually good at keeping an eye out for security issues in code proactively, so things slip through the cracks (whether human-written or not), just at a much accelerated pace.
That study is from 2022, with at most a last update 2023? (if I read it right)
I'm not at all saying that "oh 3 weeks ago, discard", but three years ago absolutely has a "before and after".
Three years ago AI generated code could basically not be used at all, and I had to rewrite just about every line. Today I have to watch out for generated code making steps in the wrong strategic and architectural direction, and over engineering, no longer hallucinating nonexisting APIs and forgetting to check the auth token.
I still look for those kinds of mistakes, but I think citing a 2022 paper for "can AI do X" is not useful.
It just means that the safeguard we put in place over decades of software engineering with humans also need to remain in place when engineering with agents.
Having properly validated libs with separate concerns and a lot of eyeballs (and agent sensors) on the code so that it can be integrated cleanly in other products. Static code analysis for our apps. System prompts that require output to be secure, etc.
All of these practices that have been touted to be obsolete because agents can now just generate tailor made code just when and where you need, are anything but obsolete. On the contrary, they are paramount to the future of software engineering.
> Overall, we find that participants who had access to an AI assistant based on OpenAI's codex-davinci-002 model wrote significantly less secure code than those without access.
That model came out in March 2022, it's GPT-3.5 !!!
I'm sorry but it's absurd to use that as evidence for anything related to AI coding today.
You can't just hedge by saying "it's somewhat older". It's like discussing dunno a new train connection between cities and citing a paper measure steam locomotives.
Hence why I said the studies here are quite old, and I'd like to see something more recent, though I guess it'll be some time before that's done.
I think most people writing code with AI nowadays will find it's still lacking in some areas. Not because the model isn't good (definitely miles ahead than what we had access to in 2022-2023), just because it wasn't actively mentioned and so the model skipped over it. The same will apply to security.
I'd also love to see recent studies. It's frustrating how even new studies end up using models which are behind the frontier at the time of publishing.
AI is advancing faster than academia can keep up. Which in general shouldn't matter. But you have this pattern where a paper comes out with a conclusion that just doesn't hold anymore by the time it's read, but people use it to form/support opinions.
I agree that we can’t compare older models to newer ones, but how much code was generated during those months/years when the models weren’t good at secure code? How much is still running publicly today? It’s still worth considering the impact to the overall ecosystem.
Luckily we have newer models going back and fixing these things.
In terms of the set of all bugs, the AIs from 2022-2023 pale in comparison with the amount of bugs written by hand in previous half century, even if 2022 AIs were worse than the average programmer.
To be fair, the warning to either take down websites or patch immediately on release came from the people who've been working on this for months (or those with advance notices like CERTs), not out of nowhere. People just took it down pre-emptively since exploit attempts (whether working or not) popped up in logs way before the patch was released, but it also took forever until the fix actually landed downstream after the advisories were released.
I think people are just latching onto the post that hit the front page earlier today about it being non-trivial to exploit, combined with the fact that there seem to be about a billion PoC out there, most of which don't work.
Right, that's what's worrying me. It is not interesting that the most widely circulated public POC/demos don't work. All you should care about is the (currently) non-public POCs that do work, of which there are apparently several.
Say what you will about OpenAI, Wiz etc but if OpenAI, Wiz, Cloudflare, the PHP Foundation and pretty much every CERT out there together recommend to take down PHP apps and patch immediately then I'm very much inclined to follow that advice.
Due to the insane media coverage there'll of course be a deluge of PoCs, most of which won't work. But constantly refreshing Twitter to see whether there's a working PoC is not a good use of time nor a great security strategy.
I think this is just false? Are we playing some kind of weird game of telephone? Is the term "direct exploit" doing way more work in that sentence than you mean it to?
The alternative route through the web forms requires the absence of ASLR. So you need to spam the endpoint, like a lot. You can probably mitigate at the edge (CDNs) or perhaps at the reverse proxy level (NGINX) through e.g. rate limiting. More of a web version of row hammer then some sort of trivial SQL injection attack that works consistently.
Maybe a LAMP stack on an extremely ancient version of Linux is vulnerable out of the box. Quit the fear mongering please, you do the security community no favors.
Are they, though? I seriously doubt some sysadmin is going to look at this and say "Maintainers said to patch, but clearly _this guy_ knows better, so I won't."
Besides, he's got a point. The initial PoC that kept popping up on Mastodon (the ones he linked in the post as well) simply didn't work. One of them even posted an updated exploit later that still didn't work, but supposedly meant the vulnerability went from CVSS 10 to CVSS 4.
Over the last few months, I invested pretty heavily into setting up SSO for my self-hosted stack. However, some apps gate SSO behind paid plans, meaning accounts and thus MFA need to be set up outside of the IdP. This leaves me with a weird mix of accounts handled by the IdP and those that I need to track manually.
There wasn't really a good way to track this mixture, especially once you add multiple MFA devices and sensitive data like recovery codes, because the IdP has one half and the password manager has the other (+ maybe it's the second factor itself).
That's why I built RecoveryCodes. It allows me to track all my accounts, see which MFA devices I've enrolled (I usually enrol 2 per account, in case one is lost) and also gives me a place to store recovery codes (instead of copy pasting them into a random .txt file).
Obviously I'm n=1, so I'm interested in hearing from slightly larger companies how you're dealing with this mix? Spreadsheets maybe, or are most of the accounts covered by IdPs and the rest don't really matter?
Does Cursor allow you to blacklist certain commands as well? I know OpenCode has this, where you can both whitelist things like grep/ls but then also blacklist things like cat .env, or rm -rf. I usually copy paste my configs nowadays, so I very rarely get prompted for any permissions (except when using Claude, which somehow decided that all users live in ~/Users, even on Linux)
reply