I've been building a cybersecurity platform (called ZeroQuarry) and regularly benchmark new models against a pretty wide set of security both code review tasks and live application pen testing tasks, and this one far underperforms relative to GLM 5.3 from every test I've done for real-world cybersecurity tasks
I think you need to catch a lot of it before it becomes a PR. Spend a lot of time teaching the agents how you code in your prompts, AGENTS.md files, etc. Make sure they know what your desired architecture is, how you want them to talk to you, how to identify and speak to their perceived blast radius, what kind of documentation you like, and so on. So when something an agent wrote does come to your desk for review, you can optimize your time/impact. I find it's a lot harder to get an understanding after the fact.
After that, I suspect a lot of people already are familiar with the https://github.com/cursor/plugins/tree/main/pstack skills, but in case you aren't they can be a good starting point. The `blast-radius`, `how`, `interrogate`, and several of the `principle-` may be interesting if you haven't already looked at this.
There are some types of reviews I find models don't do a great job at: architecture reviews (things can get too complex for them to grasp), security reviews (they get cagey thinking you're a wrong-doer), UI/UX design (they're getting better, but still most models are pretty bad). There are some dedicated tools in several of these types of "models are bad at it/unwilling" areas if you find you're often finding 1 particular part of reviews difficult and pre-prompting isn't helping.
I don't think anything new here is actually all that novel: the only thing that's changed is there used to be more of a skills and time gap than there is now.
IANAL, but I've worked with a lot of lawyers in my life on software licenses on technical products and am also currently building a platform (ZeroQuarry) which uses some decompilation/disassembly tools (not for rebuilding source software, but rather to find underlying vulnerabilities).
Tools for reverse engineering have existed for a long time (Ghidra, IDA Pro, cavaj, and a bunch of other language-specific tools), which is why most commercial companies producing non-OSS executables have put in caveats that you can't "reverse engineer or decompile, decrypt, disassemble or otherwise reduce the Software to human-readable form".
Some of these tools and underlying source code/compilation mechanisms required more skill/time than others, and now with AI, just like in software development, multimedia creation and editing, etc, the barrier to entry and the skills needed to produce something have collapsed to the point that many novices can at least "get started" at virtually no cost. A lot of people on the "builder" side have reached for obfuscation to make it more difficult (a suite of tools has been available there for a long time as well), but all that does is raises the cost of decompilation, and that cost is on an exponentially decreasing cost curve: it was never really a viable defense against "advanced" actors: the only defense against them was legal guards.
To me, I think the more interesting legal conundrum we're entering isn't pure decompilation/deobfuscation/disassembly, but rather the "clean room" carve out. Several big tech companies (Compaq vs IBM, NEC vs Intel, etc) have gotten away with using 1 team to decompile a piece of code and then have that team write a spec for a clean-room team to re-implement without looking at the decompiled software. That was deemed legal over several court cases, and now it's effectively trivial as well. Or just have an AI with vision capabilities click every button in a UI, see how they work, build the spec from there, and you don't even need decompilation tools, as we're now seeing with the Adobe suite of products: https://getartcraft.com/apps and many others
I honestly can't think of a real protection for these companies/products except signed/encrypted software at the lowest level, but that only protects companies against non-interactive software (or the non-interactive components). Some people might be old enough to remember that some software used to ship with physical devices that were the physical keys used to allow encrypted software to run. Maybe some of that makes a return?
It's an approach to defensive security for software products (especially smaller companies) by acting as the security team you don't have/can't afford. It does security recon/analysis, etc via AI.
I got tired of all of the LLM labs building firewalls of "oh, nobody is allowed to do security research/find+fix vulnerabilities in their software unless they apply for special registration."
Instead of saying "oh, anything that the model determines is security research is a vulnerability threat", it does reliably enforceable analysis like "look for a TXT record in DNS the same way a SSL provider would ensure you own the property". So it can do "live" penetration tests against your infrastructure if you provide authorization
It handles things like incoming "security researcher" e-mails to cut down on the noise of nonsense vulnerability reports by acting as your security team that defends against the reports
It provides provable/signed attestation that a pentester has checked your code/live infrastructure/APIs/etc and validated them, and/or has done a check after you've remediated whatever issues that were found. It helps all 3 sides of the "company needs pentester" and "pentester" and "auditor"/"customer" to come to agreements on what's important and what's been solved
Author here. I built ZeroQuarry, and I was considering deploying an open source link shortener or using a SaaS one. I googled around and then found iShortn, so ran the iShortn scan with it.
I found a bunch of vulnerabilities, which I sent to the maintainer and have now been patched, but some of the most interesting ones I found were that many of these vulnerabilities were actually crafted by (or at least reviewed by) CodeRabbit.
I think there are a lot of reasons to use AI code review tools these days, and no problem with CodeRabbit, but one of the things I've found interesting is a discussion from investors and potential customers about "why would I use a security code reviewer when I have an AI code reviewer in place already". I thought some of the examples here may be interesting for others.
It's a way to augment small/overloaded security teams. It can pentest and then generate a pentester-style PDF report for auditors and procurement, triage incoming e-mails from security researchers by then checking whether the vulnerabilities they claim actually exist and are exploitable, hook into GitHub to scan for vulnerabilities and auto-propose fixes or file GitHub issues for you.
It's free for Open Source projects, if anyone here is maintaining one
Also, a "just for fun" project: https://drifttrip.connelly.casa/ . I was always enthralled with the idea of taking a virtual road trip and then loading the local council's tourism promo videos at each "stop"
Yeah, I think so. It's running on a pretty small VPS and I never implemented any caching. Should have thought about that before posting I guess. I see the CPU is currently pegged. I was able to get it to load at least 1 trip myself though, so maybe retry in a bit
I've been building a product (https://zeroquarry.com) that can use a variety of models for finding vulnerabilities. One of the things I've noticed is that the models will nearly always comply with some of this, but how you prompt it matters a ton. I've worked on a set of prompts and approaches which rarely get flagged
What we've actually seen is a couple things that make this impractical "to just share a prompt". First, that nearly every major model still hallucinates a lot of vulnerabilities. Especially with temperature=0.7 as states in the original blog here, you get very inconsistent results regardless of the prompt, but that's almost kind of moot to the bigger picture. What you really need is to override the planning phase beyond asking a model "find the vulnerabilities" and you need to add another 1+ checking phases for "validate these vulnerabilities." Without that, even with the absolute best models with the highest levels of thinking enabled, you end up with garbage.
Setting the prompts and the flow with a coordinator agent directly gives a system much better capability to investigate security issues because it doesn't rely on 1-shotting things
I've been a long-timer Obsidian user with a number of plugins. Recently I launched ZeroQuarry (a product to scan code for security vulnerabilities) and pointed it at a number of Obsidian plugins. I was initially surprised to find out that so many of them had RCEs baked in: that if you open a malicious .md file, you could inadvertently run untrusted code.
I've reached out to a number of the Obsidian plugin maintainers for responsible disclosure to let them know about the issues and how to fix them, and what surprised me even more was that the most common response was roughly "yeah, we all know Obsidian plugins are basically unsafe when used against untrusted markdown content." I was surprised by this response as an Obsidian user with a number of plugins installed. It made me rethink how I think about plugins.
I like their new community program that attempts to identify some risks, but IMO it's just far too little. Obsidian really needs to have a sandboxed system. I've reached out to Obsidian as well to flag some of these risks and suggested a sandbox system as well, but haven't really had much progress in moving the needle, so I wanted to raise awareness here.
As far as I can tell, every issue you flagged in this article is now automatically caught in the new plugin review system launched last week. The new system prevents plugin updates from being released/downloaded if any of these issues are present.
The team is also working on adding permissions and more controls, see the recent announcement and HN discussion:
Since last week hundreds of plugins have been updated to patch vulnerabilities. That said there is a lot more to do and we're actively working on it. It's a very high priority.
If there are any other checks you think we should add to the automated review system I'd be happy to look into those. Since the review system is mostly open source you can also contribute to it directly, though perhaps that would be in conflict with the purpose of your company since our approach doesn't use AI for now?
I just left a job for a German B2B software company which sold primarily to large automotive, defense, and aerospace companies. Several of our customers specifically banned anything with the word "DeepSeek" -- hosted or self-hosted.
There's still a lot of naivety on what the difference is between models and platforms, and its easier for a lot of these big companies to just make a blanket statement like "nothing DeepSeek" than for their procurement teams to try to understand and negotiate with each vendor. They don't see the potential benefit over the potential risk of somebody misinterpreting or getting it wrong, so they outright ban it.
Most people that approve or buy software simply also just don't understand how models are being trained or if it's possible/how far a model could go to "introduce backdoors." A backdoor could be, from a business perspective, a model which has been trained to give answers that could hurt western business in a "strict text mode" or produces payloads in a programmatic mode that are intentionally trained to introduce software vulnerabilities.
Anyone can make arguments against these for a variety of reasons (looking at the transparency of both sides and comparing, etc) but for many reasons today and for better or worse, many Chinese models are being banned on big software contracts, which gets back to the title of the article
Thing is these models can also be a propaganda machine whether you run it locally or not. This is true no matter the origins. Chinese LLMs will never shit-talk CCP, and it will always give a rosy depiction of the Chinese government. It's perfectly understandable if companies don't want things like that. US/EU models have these problems too, but at least there are some ways to fight that: with a lawsuit or a megaphone on social networks. With Chinese models there is nothing you can do.
I've been running my own security scanning software (disclaimer: now starting a company @ zeroquarry.com) for this, and from what I've seen there's a huge value in prompts + adversarial LLM review. Without adversarial review, you get garbage (as this blog points out: 4/5 basically are nonsense) and with a good prompt, you can use almost any "near frontier" model from my experience as long as the prompt helps with the guardrails or the model doesn't protect in such a strict way
reply