Check out the approach here: https://github.com/allenporter/home-assistant-datasets and the reports/ directory has a leaderboard for function calling. I'm curious to see how well this model does.

Oh nice, I thought about doing something Ng similar with a docker image and fake devices but never got around to it

I've found this dataset specifically for Home Assistant with over 32k examples: https://huggingface.co/datasets/acon96/Home-Assistant-Reques...

Funnily, I did fine tune Qwen 1.7B with this but learned this dataset is meant for the authors extension homeLLM