Companies mass crawling don't use LLMs for crawling itself, that would be too expensive.

Make sense, but doesn't necessarily have to be an llm, just a regular dom parser will be able to tell whether an element is visible or hidden.