> The device should ideally have some kind of secret material derived per device

Thats the wrong way to do it, just require the user create a secret on first boot, and have factory reset functionality for when you forget it.