Crawl Budget and Indexation
Google gives every site a crawl allowance. Waste it on redirect chains and filter URLs, and your money pages wait in line.
Sitemaps that only list canonical URLs, redirect chains collapsed to one hop, orphan pages found and rehomed, faceted navigation stopped before it becomes a million junk URLs.
Crawl budget is real anywhere there is waste
The industry says crawl budget only matters for huge sites. In practice, any site with redirect chains, orphan pages, parameter duplicates, or a polluted sitemap makes Google work harder than it should, and Google responds by crawling less often and trusting less. The symptoms show up as slow indexing of new pages and stale versions of updated ones.
The four wastes we fix first: redirect chains that burn a request per hop, orphan pages discovered by accident with no internal links, faceted navigation generating thousands of near-identical URLs, and XML sitemaps listing redirected, noindexed, or canonicalized URLs that contradict what the sitemap claims is important.
Sitemaps and robots.txt: the two crawl signals you fully control
A sitemap is a promise: these are the canonical, indexable, 200-status URLs worth your time. Every redirected, noindexed, or duplicate URL in the file breaks that promise a little, and crawlers deprioritize files that lie to them. Keep it under 50,000 URLs, keep lastmod honest, and resubmit after structural changes.
robots.txt is the other lever: block what should never be crawled at all (sort parameters, session IDs, internal search results) and never block what you want indexed. A monthly habit of fetching both files and reading them with fresh eyes catches most crawl problems before Search Console does.
# Are any sitemap URLs redirecting or erroring? (sample the file)
curl -s https://yoursite.com/sitemap.xml | grep -o '<loc>[^<]*' | sed 's/<loc>//' | head -20 | while read u; do printf '%s -> ' "$u"; curl -s -o /dev/null -w '%{http_code}
' "$u"; done
# Read the robots file the way a bot gets it
curl -s https://yoursite.com/robots.txtRedirect chains and orphan pages
A 301 that lands on another 301 costs crawl budget, dilutes the signal, and adds latency for every user. The rule is one hop from old URL to final URL, always, and the test is one curl flag away. Chains accumulate quietly after redesigns and migrations, which is why we crawl for them quarterly on every property we manage.
Orphans are the mirror problem: real pages with no internal links, found only through the sitemap or old external links. Crawlers evaluate them without context and usually rank them accordingly. Every money page should be reachable in a small number of clicks from the homepage through real anchor links, not JavaScript click handlers.
# -L follows the chain; -w prints each hop's status and URL
curl -s -o /dev/null -L -w '%{url_effective} | %{http_code} | %{num_redirects} redirects
' https://yoursite.com/old-url
# num_redirects should be 0 for live URLs, 1 for properly redirected ones
# 2+ means a chain: point the first rule straight at the final URLFaceted navigation: the combinatorial explosion
An ecommerce site with 1,000 products and filters for size, color, price, and material can generate a million crawlable URLs, almost all of them worthless. This is the single largest source of crawl waste on catalog sites, and the fix is a crawlability matrix decided before the crawler finds the mess: which filter levels earn an indexable page, which get canonicalized to the parent, and which get blocked entirely.
The matrix is a business decision, not a technical one. Broad categories and high-demand filter combinations with real search volume get indexable pages with unique H1s. Granular multi-filter combinations get canonicalized up. Sort orders and session parameters get blocked in robots.txt. Write it down, implement it, and audit against it quarterly.
/womens-shoes index, follow (broad category)
/womens-shoes/running index, follow (subcategory)
/womens-shoes?color=red index, follow (volume + unique H1)
/womens-shoes?size=9&width=wide canonical to /womens-shoes
?sort=price_asc Disallow in robots.txt
?session_id=... Disallow in robots.txt
What you get
- Crawl waste report (chains, orphans, parameters)
- Sitemap and robots.txt rebuild
- Faceted navigation crawlability matrix
- Log-file analysis for real bot behavior
Also within Technical SEO
Website Migration SEO
Redesigns, replatforms, and domain moves without the ranking collapse: redirect mapping, parity checks, and launch-week monitoring run as a discipline, not a prayer.
ExploreJavaScript SEO
React, Vue, and SPA sites that Google and AI crawlers can actually read: prerendering, server injection, and render-budget fixes from a team that ships on this stack daily.
ExploreThe Technical SEO Checklist
The exact 40-point checklist we run on every engagement, published: crawl, render, index, schema, speed, and the header mistakes that silently kill analytics.
ExploreCanonicalization and Hreflang
Canonical tags, alternate language clusters, and the guard scripts that keep a consolidation layer from quietly telling Google your whole site lives on someone else's server.
ExploreCore Web Vitals (LCP, INP, CLS)
Field-data performance work against the 2026 thresholds: LCP under 2.5s, INP under 200ms, CLS under 0.1, verified in CrUX and Search Console, not just Lighthouse.
ExploreAI Crawler Access Policy
Training bots and search bots are different crawlers now. Block one group, welcome the other, and stop losing AI citations to a robots.txt file nobody has read since 2023.
ExploreStructured Data and Schema
The schema that still moves the needle in 2026: Organization with sameAs chains, Service, Article, and FAQ. Entity clarity for Google and the answer engines, validated in the served HTML.
ExploreQuestions we hear about crawl budget and indexation
The budget itself rarely does, but the hygiene behind it does. Redirect chains, orphan pages, and a polluted sitemap hurt trust and indexing speed at any size. Small sites feel it as new pages taking weeks to index instead of days.
Not ready for a retainer?
Start with the Growth Audit: $1,500, flat.
A senior review of your site, search visibility, AI-search readiness, and paid spend, delivered as a prioritized 90-day roadmap in 10 business days. Fully credited toward your first month if we keep working together.
Fixed price · fixed date · credited forward
Before you scroll away
Booking takes sixty seconds. Here's what happens next.
01
Pick a time
Thirty minutes, video or phone, your agenda. The booking form takes about a minute.
02
Get the benchmark
We audit your position before the call: search rankings, AI answers, competitors. You keep the findings either way.
03
Decide with numbers
A scoped plan and one number to hold us to. No obligation, no retainer pitch on a first call, no drip campaign after.
Audit my crawl budget
One conversation, one plan, one number to hold us to. Book your free consultation today.
Prefer to write first? Send us a message · Email [email protected] · Call (727) 280-6953
