So what about flight searches where we have to query several 3rd party providers, and can take 45 seconds to get results from all of them (out of my control). I can dynamically update the page (and do) but a scraper would have to wait 20-45 seconds to get the 'cheapest' flight from my site. I can add async the queries and have the fastest pipes, but if the upstream providers take their time (they need to query their GDSs as well), there's not much you can do.
Oof, reminds me of the data pipelines I maintain that pull data from Salesforce. I've optimized the heck out of all of our internal database queries and they're blazing fast. Then my pipeline has to wait patiently for the SOQL response.
