Crawl Budget and Indexation

Google gives every site a crawl allowance. Waste it on redirect chains and filter URLs, and your money pages wait in line.

Sitemaps that only list canonical URLs, redirect chains collapsed to one hop, orphan pages found and rehomed, faceted navigation stopped before it becomes a million junk URLs.

Crawl budget is real anywhere there is waste

The industry says crawl budget only matters for huge sites. In practice, any site with redirect chains, orphan pages, parameter duplicates, or a polluted sitemap makes Google work harder than it should, and Google responds by crawling less often and trusting less. The symptoms show up as slow indexing of new pages and stale versions of updated ones.

The four wastes we fix first: redirect chains that burn a request per hop, orphan pages discovered by accident with no internal links, faceted navigation generating thousands of near-identical URLs, and XML sitemaps listing redirected, noindexed, or canonicalized URLs that contradict what the sitemap claims is important.

Sitemaps and robots.txt: the two crawl signals you fully control

A sitemap is a promise: these are the canonical, indexable, 200-status URLs worth your time. Every redirected, noindexed, or duplicate URL in the file breaks that promise a little, and crawlers deprioritize files that lie to them. Keep it under 50,000 URLs, keep lastmod honest, and resubmit after structural changes.

robots.txt is the other lever: block what should never be crawled at all (sort parameters, session IDs, internal search results) and never block what you want indexed. A monthly habit of fetching both files and reading them with fresh eyes catches most crawl problems before Search Console does.

bashThe 60-second sitemap and robots sanity check
# Are any sitemap URLs redirecting or erroring? (sample the file)
curl -s https://yoursite.com/sitemap.xml | grep -o '<loc>[^<]*' | sed 's/<loc>//' | head -20   | while read u; do printf '%s -> ' "$u"; curl -s -o /dev/null -w '%{http_code}
' "$u"; done

# Read the robots file the way a bot gets it
curl -s https://yoursite.com/robots.txt

Redirect chains and orphan pages

A 301 that lands on another 301 costs crawl budget, dilutes the signal, and adds latency for every user. The rule is one hop from old URL to final URL, always, and the test is one curl flag away. Chains accumulate quietly after redesigns and migrations, which is why we crawl for them quarterly on every property we manage.

Orphans are the mirror problem: real pages with no internal links, found only through the sitemap or old external links. Crawlers evaluate them without context and usually rank them accordingly. Every money page should be reachable in a small number of clicks from the homepage through real anchor links, not JavaScript click handlers.

bashOne-hop rule: trace every redirect to its final URL
# -L follows the chain; -w prints each hop's status and URL
curl -s -o /dev/null -L -w '%{url_effective} | %{http_code} | %{num_redirects} redirects
'   https://yoursite.com/old-url

# num_redirects should be 0 for live URLs, 1 for properly redirected ones
# 2+ means a chain: point the first rule straight at the final URL

Faceted navigation: the combinatorial explosion

An ecommerce site with 1,000 products and filters for size, color, price, and material can generate a million crawlable URLs, almost all of them worthless. This is the single largest source of crawl waste on catalog sites, and the fix is a crawlability matrix decided before the crawler finds the mess: which filter levels earn an indexable page, which get canonicalized to the parent, and which get blocked entirely.

The matrix is a business decision, not a technical one. Broad categories and high-demand filter combinations with real search volume get indexable pages with unique H1s. Granular multi-filter combinations get canonicalized up. Sort orders and session parameters get blocked in robots.txt. Write it down, implement it, and audit against it quarterly.

textThe crawlability matrix, decided on purpose
/womens-shoes                      index, follow   (broad category)
/womens-shoes/running             index, follow   (subcategory)
/womens-shoes?color=red           index, follow   (volume + unique H1)
/womens-shoes?size=9&width=wide   canonical to /womens-shoes
?sort=price_asc                   Disallow in robots.txt
?session_id=...                   Disallow in robots.txt
Client dashboard preview: spend by category with manual budget tracking
From the client dashboard, every engagement reports on a view like this, with a documentation library to help you read it.See a live sample

What you get

  • Crawl waste report (chains, orphans, parameters)
  • Sitemap and robots.txt rebuild
  • Faceted navigation crawlability matrix
  • Log-file analysis for real bot behavior

Questions we hear about crawl budget and indexation

The budget itself rarely does, but the hygiene behind it does. Redirect chains, orphan pages, and a polluted sitemap hurt trust and indexing speed at any size. Small sites feel it as new pages taking weeks to index instead of days.

Not ready for a retainer?

Start with the Growth Audit: $1,500, flat.

A senior review of your site, search visibility, AI-search readiness, and paid spend, delivered as a prioritized 90-day roadmap in 10 business days. Fully credited toward your first month if we keep working together.

Book the Audit

Fixed price · fixed date · credited forward

Before you scroll away

Booking takes sixty seconds. Here's what happens next.

Book My Free Consultation

01

Pick a time

Thirty minutes, video or phone, your agenda. The booking form takes about a minute.

02

Get the benchmark

We audit your position before the call: search rankings, AI answers, competitors. You keep the findings either way.

03

Decide with numbers

A scoped plan and one number to hold us to. No obligation, no retainer pitch on a first call, no drip campaign after.

Audit my crawl budget

One conversation, one plan, one number to hold us to. Book your free consultation today.

Prefer to write first? Send us a message · Email [email protected] · Call (727) 280-6953