Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
jhwhite
on June 30, 2014
|
parent
|
context
|
favorite
| on:
Robots.txt Disallow: 20 Years of Mistakes To Avoid
I haven't read the entire article but we were discussing this at work a few weeks ago. You can set the crawl delay in Google Web Master Tools but they only adhere to that setting for 90 days then they go back to their default.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search: