Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
sschueller
on July 4, 2014
|
parent
|
context
|
favorite
| on:
Google.com/killer-robots.txt
February 1994[1], so the 20 year anniversary of the robots.txt file.
[1]
http://en.wikipedia.org/wiki/Robots_exclusion_standard
Consider applying for YC's Winter 2027 batch!
Applications
are open till November 2.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
[1]http://en.wikipedia.org/wiki/Robots_exclusion_standard