Crawler

A bot (also called a spider) that follows links across the web to discover pages and feed them into a search engine's index. Googlebot, Bingbot, and other crawlers determine what content appears in search results.

PreviousCrawlability NextCustomer Journey

Related Terms

Cached Page

A snapshot of a web page stored by a search engine from its most recent crawl. Checking the cached version lets you verify what Google actually sees — helpful for diagnosing indexing issues or cloaking problems.

Canonical Tag

An HTML hint (rel="canonical") placed in a page's head section that tells search engines which URL should be treated as the primary version. Canonical tags prevent duplicate content from splitting your ranking signals across multiple URLs.

Canonical URL

The single URL that search engines recognize as the authoritative version of a page when similar or identical content appears at multiple addresses. Setting the correct canonical URL consolidates link equity and avoids index bloat.

Cloaking

Serving one version of a page to search engine crawlers and a different version to human visitors. Cloaking is a clear violation of Google's guidelines and can result in a manual penalty or complete removal from the index.

Co-citation

When two pages are repeatedly mentioned or linked together by unrelated third-party sources. Search engines may infer a topical relationship between co-cited pages even if they never link to each other directly.

Co-occurrence

The pattern of certain words or phrases appearing together across many documents about the same subject. Search engines use co-occurrence data to understand semantic relationships between terms and topics.

Put this into practice with KarmaLinks, an AI backlink exchange platform where verified websites trade do-follow backlinks through safe ABC-only triangular exchanges.