I’m building Hammer Labs (https://hammer.ai) to study when healthcare AI agents are wrong and when they should refuse to answer (judgement).
It started after spending 15 years building AI for insurers, hospitals, data companies, and startups. Almost every system ended with "a human reviews the output". That person was usually a nurse, medical director, or certified coder. These are some of the hardest people to hire, and the same people automation was supposed to help.
The problem is that real claims do not have an answer key. You cannot reduce human review until you can measure when an agent is wrong.
Getting claims data is also difficult. It can take a year of data agreements, privacy reviews, and procurement. Even then, you may not know what the correct decision should have been.
So we generate claims. Utilization and case mix come from published data. Claims are priced using real fee schedules and contract terms. Payers behave differently, like real payers do. We plant errors on purpose, so the correct answer exists before any model runs.
On top of that, we are building benchmarks for overreach, refusal, errors by record type, and detection time. We are also building small MCP tools that refuse when evidence is missing. Every number includes its source, date, and basis.
What I find interesting is how much of this sits between actuarial work and machine learning. Both are needed, but I do not see many people connecting them.
I was trying to learn more about your project. The comment above was digestiable for me as a non-healthcare person, but the website is really difficult for me to understand.
The web design (and text?) really come off as ChatGPT written, which lowers my interest in spending time to understand it.
Unity is useless for web games. Period. I have the scar tissue to prove it. The whole asset management pipeline is just insanely hard to use. Can’t collaborate or even source version control across devs. Compilations suck. The whole box in a box is dumb. It wastes real estate. So much insane friction. Just because of 3d artists and fan boys the whole eco system is a waste of time. Open web tools with hot loading for devs is yeaaars ahead of Unity. And their performance will always suck.
Amazing work Arjun & Anirudh! I know you have been working on this for a while! What has been your biggest lesson learned building such a sophisticated devtool?
Thanks! The biggest lesson for us has been in finding the balance between being opinionated v/s being flexible in the abstractions we build. I think it's easy to go too far in either direction but we're learning as we build about how to make these decisions better.
For example, there were 2 different versions of the sandbox creation APIs that we built in the past - one that optimized for docker image-driven workflows, and another that designed for the sandboxes being immutable, but our current APIs are document-style and mutable, giving additional flexibility to build more use-cases on top like feature testing.
I work on a product called smartomix.com part of the doc.ai (recently acquired by Sharecare). We help researchers do just this. DM if you think this would be useful for you.
Hey, co-founder of HOTG.ai here! Nice to see how folks are getting more interested in WASM and deployments. We hadn't specifically targeted the serverless cloud area but we are considering it. We are an edge first company (mobile web, mobile, embedded linux etc). Are you using WASM in production today? If not, why not?
Hi: Yes, we use WASM by the way of Deno-Deploy and Cloudflare Workers but for web & logging workloads. We do not have a specific need for WASM but it speed: It is faster than any other serverless sandboxes (Kata/gVisor/Firecracker/Docker) out there. TinyML is something I keep a keen eye on as that's likely going to be what we would next use serverless for and prefer if it was WASM-able.
Stealth Startup | REMOTE, SF bay area | Multiple Engineering Role | Part time & contract
We are an early stage startup working on light weight containerization and virtualization open source technologies for constrained devices. We are looking for software developer with experience: Rust, C/C++, Embedded Systems, MbedOS, Arm, iOS and Android.
Please contact holly ‘at’ hotg.ai
Stealth Startup | REMOTE, SF bay area | Multiple Engineering Role | Part/Full time & contract
We are an early stage startup working on light weight containerization and virtualization open source technologies for constrained devices. We are looking for software developer with experience: Rust, C/C++, Embedded Systems, MbedOS, Arm, iOS and Android.
It started after spending 15 years building AI for insurers, hospitals, data companies, and startups. Almost every system ended with "a human reviews the output". That person was usually a nurse, medical director, or certified coder. These are some of the hardest people to hire, and the same people automation was supposed to help.
The problem is that real claims do not have an answer key. You cannot reduce human review until you can measure when an agent is wrong.
Getting claims data is also difficult. It can take a year of data agreements, privacy reviews, and procurement. Even then, you may not know what the correct decision should have been. So we generate claims. Utilization and case mix come from published data. Claims are priced using real fee schedules and contract terms. Payers behave differently, like real payers do. We plant errors on purpose, so the correct answer exists before any model runs.
On top of that, we are building benchmarks for overreach, refusal, errors by record type, and detection time. We are also building small MCP tools that refuse when evidence is missing. Every number includes its source, date, and basis.
What I find interesting is how much of this sits between actuarial work and machine learning. Both are needed, but I do not see many people connecting them.
25 published refusals: https://hammer.ai/worlds/refusals/ .
Runs on rate and policy evidence https://hammer.ai/reimbursement-evidence/ and savings claims https://hammer.ai/savings-claims/ .
AgentPlugin is Apache-2.0: https://github.com/hmmrlabs/hammer-plugin