I've been working on a tool (https://dynobox.xyz) that acts as a deterministic integration test / behavioral test layer for some of the skills i've been working on / sharing.
It feels like a full eval suite is a bit heavy handed and really all I care about is if certain files are touched / left alone or if my skill is actually read. The tooling has much more functionality built in if you want to check it out!
For skill files / prompts I share I make sure that I use the cross harness functionality since I use codex but a bunch of my coworkers use claude (and then one using antigravity...)
This is really great! I have a few personal skills I've been maintaining for generating and rendering mermaid diagrams but the overall coverage here is way beyond what I was doing. ++ to visuals being a key missing tool in a lot of these harnesses.
Not sure how you're testing the output of this but I posted about my tool i've been working on earlier today https://news.ycombinator.com/item?id=49274758 and spun up a quick (very basic) test suite example for this:
It’s interesting to see eval frameworks start supporting specific harnesses. Once you’re testing the same skill across Claude Code, Codex, and others, the harness starts to feel like another dimension in the test matrix. It reminds me of browser compatibility testing.
I've been working on https://dynobox.xyz which is a local test runner for agents skills.
I was working on an agentic payments (MPP) powered service and had a hard time getting consistent behavior from my skill.md i was hoping to distribute. so like usually, one project let to another...
If anyone wants to try it out and has feedback I'd love to hear!
ive been experimenting with pointing some custom opencode agents and commands specifically at local models for really small things like commit messages which definitely don't need an LLM
I've been working on a tool (https://dynobox.xyz) that acts as a deterministic integration test / behavioral test layer for some of the skills i've been working on / sharing.
It feels like a full eval suite is a bit heavy handed and really all I care about is if certain files are touched / left alone or if my skill is actually read. The tooling has much more functionality built in if you want to check it out!
For skill files / prompts I share I make sure that I use the cross harness functionality since I use codex but a bunch of my coworkers use claude (and then one using antigravity...)
reply