Author here. In short:

- We launched an AI bug scanner 6 months ago. We wanted to know what the bugs that get fixed have in common.

- We clustered 1,000 bugs from 99 codebases and grouped them by failure mechanism.

- 21 recurring mechanisms cover 70% of the bugs. These same mistakes show up again and again in completely unrelated products. We seem to be writing similar bugs over and over again with agents.

- A common denominator is silence. Bugs that make it to production today are not easily ‘legible’. They don’t crash, they pass CI, they look fine. You don’t see them unless you look for them.

If you’d like to know anything else about these bugs, let us know. Happy to analyze the dataset further.

interesting data and results. it kind of looks like they are good at coding plainly no buffer overflows or such things, but as noted application logic is the trick. Things that are difficult for humans (authentication, paralelism) seems also tricky for them.

Id wonder if they are badly putting together the logic from good instructions or that the prompting was missing pieces and it followed correctly but provided some broken code due to missing requirements / details.

Good question. We don't collect the 'what lead to this code' data, such as agent traces, so I can't say for sure.

From my own experience, I would attribute most of bugs that I write to the agent not considering one of the many constraints it should consider. This could be it assuming a different shape of data, or not browsing the codebase thoroughly enough at some point during its work and missing a parallel, or it not testing an edge case because it didn't think of it, and so on.

So, I would place this more under the latter, 'missing requirements / details', bucket. But its not clear to me that I should be giving the requirements as a prompt. Similarly to a good engineer, an ideal agent would spend a good amount of time understanding all the requirements before completing work. This could include browsing the codebase more extensively, running more tests, asking me for details it cannot find, and so on.

Not surprisingly as the AI is trained on code written by humans, so things that are difficult for humans will also be difficult for them to catch

I would love to do this on my codebase after every commit

Some names are just too tempting https://arxiv.org/abs/1507.02672

This looks promising! Does any LLM understand Instantdb yet?

For comparison, manslaughter average sentence in Finland seems to be around 9.5 yrs

You can get the ball off the ground if you turn your phone upside down. Then you can just sort of fly over the places where you'd normally fall. Takes maybe a minute or two to complete all the levels.

Ive enjoyed the classical sales books like SPIN selling, Challenger sale, Fanatical prospecting, The Psychology of Selling.

Has anyone experimented with integrating real-time lipsync into a low-latency audio bot? I saw some demos with d-id but their pricing was closer to $1/minute which makes it rather prohibitive

Interesting project, thanks for sharing

Knowing when to speak is actually a prediction task in itself. See eg https://arxiv.org/abs/2010.10874

Would be indeed great to get something like this integrated with whisper, LLM and TTS

Hard for me to imagine that this could be solved in text space. I think the prediction task needs to be done on the audio.

We thought about doing this in Whisper itself, since its already working in the audio space.

Yes, this is something we want to look into in more detail, really appreciate sharing the research.