Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

My search engine has some not-very-obvious tools for exploring the link graph, that will occasionally turn up interesting things.

Similarity navigation: https://marginalia-search.com/site/simonwillison.net

Backlinks: https://marginalia-search.com/site/simonwillison.net?view=li...



Out of curiosity, I see my website not being well indexed and I wondered whether it is because I include ``` User-agent: * Disallow: / ``` in my robots.txt. What should one add to allow Marginalia to crawl their website?


This should work

  User-agent: search.marginalia.nu
  Allow: /


Thank you! I just added it :)




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: