I didn't end up needing to go full scrcpy because my use case just runs Android in an emulator (Waydroid) for my Hermes agent to run apps. If I need vision or (x,y) coordinate interaction, I can just screenshot the emulator window and interact via ADB. However, I ended up using the accessibility tree to create a semantic, ephemeral ref API as the preferred mechanism for interaction. Basically more 'Playwright for Android' than 'Remote Desktop for Android'. But, unfortunately, the accessibility tree is often incomplete or inaccurate.
Would be great if you have a way to do it programmatically.
https://github.com/Genymobile/scrcpy