Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

The article contains some good observations, but I'm struggling to understand this one:

"Some sites try to communicate with Google through comments in robots.txt"

In the examples given, none appear to be trying to "communicate with Google through comments" - how is including...

  # What's all this then? 
  #   \
  # 
  #    -----
  #   | . . |
  #    -----
  #  \--|-|--/
  #     | |
  #  |-------|
...a "mistake" to avoid? There's no harm in it at all.


"Some sites try to communicate with Google through comments in robots.txt"

I thought that was the whole point of robots.txt


No, the point is to communicate with Google through non-comments in robots.txt.


I don't think those examples were of people trying to communicate with a crawler. I think they were examples of comments that the owners knew would be thrown away by crawlers.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: