Robots.txt

A plain text file in your site's root directory that tells search engine crawlers which URLs they can and cannot access. Robots.txt helps you manage crawl budget by steering bots away from low-value or private pages.

Related Terms