My search engine has some not-very-obvious tools for exploring the link graph, t... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		marginalia_nu 53 days ago \| parent \| context \| favorite \| on: Personal blogs are back, should niche blogs be nex... My search engine has some not-very-obvious tools for exploring the link graph, that will occasionally turn up interesting things. Similarity navigation: https://marginalia-search.com/site/simonwillison.net Backlinks: https://marginalia-search.com/site/simonwillison.net?view=li...

ggirelli 50 days ago [–]

Out of curiosity, I see my website not being well indexed and I wondered whether it is because I include ``` User-agent: * Disallow: / ``` in my robots.txt. What should one add to allow Marginalia to crawl their website?

marginalia_nu 50 days ago | [–]

This should work

  User-agent: search.marginalia.nu
  Allow: /

ggirelli 45 days ago | | [–]

Thank you! I just added it :)

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact