How search engines crawl, index and rank pages
The three stages every page goes through, and the files and tags that control each stage.
Most SEO mistakes come from mixing up three separate stages: crawling, indexing and ranking. If you know which stage a problem belongs to, you know which tool fixes it.
Crawling: finding the page
A crawler is a program that fetches pages and follows the links on them. It finds new URLs through links from other pages, internal links on your own site and the XML sitemap you submit.
- robots.txt tells compliant crawlers which paths they should not fetch, for example
Disallow: /cart/. - A page with no internal links pointing to it, called an orphan page, is hard to discover.
- Slow servers and long redirect chains waste crawl effort on large sites.
Indexing: understanding and storing the page
After fetching, the search engine reads the content, title, headings, links and structured data, and decides whether to store the page in its index. Not every crawled page gets indexed. Thin, duplicate or low-value pages are often left out.
- noindex in a robots meta tag asks for the page to be kept out of the index.
- rel="canonical" points duplicates to the preferred URL.
- The mobile version of the page is the one mainly used, so content must not be hidden on phones.
A common error is blocking a page in robots.txt and adding noindex at the same time. The crawler cannot fetch the page, so it never sees the noindex tag.
Ranking: choosing the best result
When someone searches, the engine picks indexed pages that best match the meaning and intent of the query. Relevance, content quality, links from other sites, page experience and freshness for time-sensitive topics all play a part. Nobody outside the search engine knows the exact weighting, so be wary of anyone who claims a fixed formula.
A quick diagnosis habit
When a page is not getting search traffic, ask these questions in order:
- Can it be crawled? Check robots.txt and internal links.
- Is it indexed? Check for noindex, canonical tags pointing elsewhere, and duplicate content.
- Does it match the intent of the query, and is it better than what already ranks?
Working from crawl to index to rank stops you rewriting content when the real problem is a blocked URL.
More in Digital marketing