My worry with the confidence scoring is that it conflates "an agent used this and didn't obviously break" with "this is correct". An agent can follow bad advice for several steps before anything fails. So a KU gaining confirmation weight doesn't tell you much about whether it's actually true, just that it propagated. You're crowd-sourcing correctness from sources that can't reliably detect their own mistakes.
It's why at Tessl we treat evals as a first-class part of the development process rather than an afterthought. Without some mechanism to verify quality beyond adoption, you end up with a very efficient way to spread confident nonsense at scale.
At first glance this looks like an entire ecosystem full of slop and by running that eval you generate more? I'm looking for something a bit more curated.
No, the context can be human created as much as it could be llm generated. The suggestions are based on Anthropic best practices and allow the agents to activate, and use the skills better, make the text clearer for the agent etc.
I submitted a package-lock.json file to the playground and got a vulnerability report after processing. The sort order next to the pie chart is weird. Medium / High / Critical / Low. I'd expect Critical / High / Medium / Low?
The vuln report ended up in my email spam folder.
I had to hit 'resend' multiple times to receive the verification email. Once I did, I had to either create a new account or login. I don't yet have a password. When I tried to create an account, it said my email was already taken. This onboarding flow seems quite janky.
Is Vulert Open Source software? I couldn't find any links or repos. What does "Join the Open-Source Security Movement" mean in this context?
I submitted two open-source tools. The submission form has a field for 'License' in which the only two options are 'Free' and 'Commercial'. Those aren't licenses. Maybe adjust that field to either say 'cost' or 'terms', or actually have a license field which lets you paste an SPDX entry (or entries) or pick a license from a list.
There certainly used to be a strong push to have internal people use the product a lot more during the development cycle. There was also a real desire to make the devel version actually usable. That fell by the wayside, sadly.
Having your developer workstation break while you have a backlog full of stuff to do, would absolutely make you less motivated to run the developer release. Especially if you're not on the desktop team.
I suspect it was a lot easier to feel like that was one's duty when the company was smaller and you were closer to every piece of it. It was also probably easier to get issues resolved when you knew exactly who to talk to. Maintaining that culture as you grow is probably quite the challenge!
reply