Crawler

A bot (also called a spider) that follows links across the web to discover pages and feed them into a search engine's index. Googlebot, Bingbot, and other crawlers determine what content appears in search results.

Related Terms