Most drivers relies on being able to do their job unencumbered. Which means giving them information they need, and relaying their messages to the material. And that usually means tight coupling to the kernel. Creating a translation layer is maintaining the Linux API (aka all current functions). And BSDs have different submodules for some stuff.
Most drivers issue stems from the hardware being undocumented. Not knowing the initialization parameters, not knowing the protocol, and the functions,as well as the various settings that influence each other. And Linux allows for binary blobs, which means things are opaque even when working.
The problem is that this only works if your hardware supports emulation and you can actually use PCI pass through - for USB devices, you'd need to pass the whole USB hub, for example.
