Robots.txt

Robots.txt is a plain text file at the root of a site that tells crawlers which paths they may fetch. It controls crawling, not indexing: a page blocked in robots.txt can still appear in results if other sites link to it. To keep a page out of results, let it be crawled and mark it noindex instead. Our technical SEO audit guide covers both.