Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

if that is any consolation, no one gives a shit about xitter's ToS either. it will continue to be scrapped by every major player.


How exactly is it being scraped? My understanding is Twitter and LinkedIn are both huge pains in the ass to scrape right now.


There's a number of companies out there, like "brightdata", which pay a small amount to app developers to install a native "sdk". That SDK mimics a browser, and makes requests as if the user's device is doing it.

Since it's using a large number of real user's devices, and closely mimicing real web browsers, it ends up looking incredibly similar to real user traffic.

Since twitter allows some amount of anonymous browsing, that's enough to get some amount of data out. You can also pay brightdata for one large aggregated dataset.

https://bright-sdk.com/

This is part of the AI revolution, user's devices being commandeered to DDoS small blogs and twitter alike to feed data to the beast.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: