That's why you can give guidance to crawlers using sitemaps. | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		Chiron1991 5 months ago \| parent \| context \| favorite \| on: URL-Driven State in HTMX That's why you can give guidance to crawlers using sitemaps.

Quarrel 5 months ago [–]

Which, just like robots.txt, worked decently well until the AI bot crawlers overwhelmed all else.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact