Not completely impossible. You could have a default deny firewall, have your DNS resolver trigger an update to allow outgoing connections to the resolved IPs, and possibly also require connections pass though an SNI-sniffing proxy that only allows domains that your DNS resolver has allowed. Essentially by default you'd be blocking all custom protocols, and you'd only allow what looks like well-behaved TLS web traffic to allowed domains to flow.
Bad traffic could flow to a "good" domain, and then you need to decide whether that domain is actually "good".
couldn't they just hide their ad endpoints behind the proxy that serves their site? I can think of multiple ways to do this that aren't very difficult. I have had to implement something in my work to get past certain adblocking behavior that was going by domain
Sure, but now you've at least made them use a more expensive L7 proxy to do it, and you can decide to block malicious actors like that entirely (blocking the "good" domain).
Yes, you can do it with an L7 proxy. You've been able to do that all along though, so I suppose there are reasons why surveillance networks prefer to not proxy through the websites that host their scripts. That has nothing to do with DoH to subvert network security monitors though.
You create a server and host it on IP x. You create a cert for it. You add the public key to your app.
Your app can now communicate with that IP over port 443 with that certificate. Remember that the idea that the domain must match the one in the certificate is a setting, enforced by the browsers. If you run your own code you can perfectly override that.
Now you can do whatever you like on that connection.
In fact, you don't HAVE to go that far. Many applications these days do private key pinning and use that connection to load the ads. IMDb does that on the iPhone.
MyQ and myBMW use the same to 'protect' the connection. MyQ's implementation of this, and subsequent implementation of CloudFlare's bot protection completely broke home-assistant's connection. All because they want you to use their app (and get bombarded with ads).
Doh/DoT was supposed to bring in MORE privacy for users, as it allowed users to resolve addresses without the system servicing the connection (ISP / StarBucks / McDonald's) from being able to see or modify the responses (think captive pages).
But all it brought was more spying. I am a firm believer that I should be able to inspect all traffic that an application sends out over my internet connection.
Some browsers, apps, or devices might let you disable DoS/DoT or might let you configure it to use your own DNS server, but none of them have to let you and even when they give you that option they can still do whatever want (https://discourse.pi-hole.net/t/chromium-bypasses-pi-hole-by...)