Median is < 1d for major ones

We did a small survey of major incidents after shai-hulud and came to 2d cooldowns through that

Awesome

I've suspected for awhile that the frontier models work this way bc from a speed + quality perspective this feels pretty intuitive

yeah for botsbench.com side where we do cybersecurity investigation evals of models + harnesses, we've been having to play with:

1) time budget model gets (5min / task) -- this became a real problem with opus/astra/fable at high+

2) tuning which effort level we report on (when not all)

It's incredible how well talking points on wedge issues get consumed, accepted, and amplified by both bots and people. And even though AI can now detect this in seconds, the problem seems worse nowadays, not better.

(Source: we help power tools used by the big boys to analyze posts like this & their underlying narrative campaigns)

GLM 5.3 Flash even more so... But yes :)

We were able to do a full port of GFQL from pandas to polars, cypher graph queries on dataframes, including both our CPU + GPU modes, and hit massive speedups: https://www.graphistry.com/blog/cypher-on-polars-cpu-gpu-gra...

It's been impressive!

Funny enough, GFQL (oss cypher on CPU/GPU dataframes, no DB needed) runs optimized graph queries with the usual CSR-like indexes... But underlying data is columnar as in duckdb. Likewise, for fast billionscale perf on single node, maps graph compute to pure vector ops. Storage, data representation, & reasoning are largely separated concepts.

If you can't tell bytes are leaving a node, you probably shouldn't be selling security services or testifying to congress you are taking the lead in AI security

My intuition is that many of the better & bigger 'private' code bases, at least in terms of claude code and codex... are not in fact private at this point.

One lesson of running botsbench.com, in a slightly different domain, is to measure for model contamination every time.

I spent a while in big tech and remember several unique patterns of internal code based. Your comment inspired me to try to coax ChatGPT into spitting out code that was inspired by proprietary, private code. Surprisingly, it did it with no problem - I referenced an idiom from a tech company and it wrote code that really would have only been relevant for that vertical. When I asked how it learned the pattern, it said “from my learned understanding of CompanyX’s internal coding conventions”. When I asked “how do you know about those internal conventions” ChatGPT said “I don’t have access to that internal code, I overstated what I know”. Internal coding conventions are the least of our worries at this point, cat is out of the bag.

You cannot introspect model training by prompting the model. Whatever answer it hallucinated on your query "where do you know that from" has almost certainly very low bearing on reality.

True. It sort of doesn’t matter how it was trained. The fact that it knew the company based on an internal idiom is super surprising

Well how do you know which statement is truthful? These LLMs confidently say they know things that they don't all the time.

much harder to do in OP's case, matching flavour then referencing that specific companies name when asked how it know to flavour this way? thats astronomically low for randomly selected plausible tokens without some data prior, like that companies codebase.

My own experience is opus being lousy at an extremely niche math task, but it was still easier for me to describe what it needed to do to get code and correct issues in its reasoning/working than to write myself. a minor model number change later and it's nailing everything, despite my opt-out. Its is astronomically unlikley others were working on this also, especially at that level, especially this application.

so, safe to say they _all_ train models on chats, the only difference being if you "opt out" you at least have some defence later when they steal your work and claim it as their models original output.

For us it's ~easy: we do security investigation CTFs against log dumps / DBs like Splunk. We run the AI CTF as usual.. except without the DB nor internet access. If they still get some right, that's a problem. (And most public CTFs are contaminated now for that reason.)

There's a funny middle where they may not know the exact answer confidently, but do know where to get them, or aspects of the problem. That's still a potential lift in the competition . So we also check the traces to see if there is any knowledge during intermediate step DB queries that it shouldt have based on the previous steps.

I gave a recorded talk at BSides Las Vegas a few weeks ago on 'adversarial evals' that gets into the kind of cheating we've been having to deal with.

Much earlier, I tried to set up some static analysis tools so that the coding agent would have access to dataflow analysis etc. tools instead of just grep for typed python. If there were benefits, they weren't easily apparent :(