Every site has a ceiling on how much attention search engines will give it. Google calls the two halves of this crawl capacity — how hard they can hit your server without hurting it — and crawl demand, which is how interested they are in your content in the first place. Multiply the two and you get the practical number of URLs Googlebot will fetch from you in a given period.
For a 40-page brochure site none of this matters. Google will crawl everything, several times over, and you can stop reading. It starts to matter the moment your URL count runs into the thousands, or your site generates URLs automatically — filters, search results, session parameters, paginated archives — because that is when the number of addresses on your site quietly exceeds the number of pages you actually wrote.
The result is crawl waste: Googlebot spending its allowance fetching things that will never rank, while your new product page waits nine days to be discovered.
How to tell whether you have a problem
Open Google Search Console and go to Settings, then Crawl stats. Two numbers matter. The first is total crawl requests over the last 90 days. The second is the breakdown by response code and by file type.
Now compare that with reality. Count the URLs you actually want indexed — your sitemap is a decent proxy. If Google is making 200,000 requests a month to a site with 3,000 real pages, the extra 197,000 are going somewhere, and it is worth finding out where.
The other signal is in the same report: the share of requests returning 404, 301 or 5xx. A healthy site is overwhelmingly 200s. If a third of your crawl budget is being spent following redirects, that is a third of your budget spent learning nothing new.
- Lots of 404s usually means internal links or a sitemap pointing at pages that no longer exist.
- Lots of 301s means internal links point at the old URL rather than the destination. Fix the links, not just the redirects.
- Lots of 5xx means your server is struggling, and Google will throttle its own crawling in response — which lowers your ceiling directly.
The four things that waste the most crawl
1. Faceted navigation
A shop with five filters, each with six options, can generate tens of thousands of URL combinations from a single category page. Each one looks like a distinct address to a crawler. This is by far the largest source of crawl waste on ecommerce sites, and it is the reason parameter handling exists.
2. Internal search results
If your site search produces a crawlable URL such as /?s=blue+widget, and anything on the web links to one, Googlebot will find it and start exploring. Search result pages are thin by definition and should never be indexed.
3. Redirect chains
Every hop in a chain is a separate request. A three-hop chain costs three times the budget of a direct link, delivers the same page, and dilutes signals along the way. Audit your internal links so they point at the final URL.
4. Endless pagination and archives
Tag archives, date archives and author archives on a busy blog can easily triple your URL count while adding nothing a reader would search for. Most sites should keep one archive type and noindex the rest.
Fixing it: what to use when
The three tools people reach for — robots.txt, noindex and canonical tags — do genuinely different jobs, and using the wrong one is how sites end up worse off than when they started.
- robots.txt stops the crawl. Use it for URL patterns that have no value at all and that you never want fetched — internal search, cart URLs, tracking parameters. It saves budget immediately.
- noindex allows the crawl but keeps the page out of the index. Use it for pages a user might legitimately reach but that should not compete in search — thin tag archives, filtered views.
- rel=canonical allows the crawl and consolidates signals onto a preferred URL. Use it where several URLs show substantially the same content and you want the ranking strength pooled.
The trap is combining the first two. If you block a URL in robots.txt, Googlebot never fetches the page, which means it never sees the noindex tag you added. The URL can then sit in the index indefinitely with no description. If a page needs deindexing, let it be crawled until it drops out, and only then block it.
Build the rules with our robots.txt generator rather than from memory. Path matching in robots.txt is prefix-based, and a missing trailing slash blocks far more than you intended.
Make your sitemap tell the truth
A sitemap is the clearest statement you can make about which URLs you consider canonical. That only works if the list is accurate. A sitemap containing redirects, 404s, noindexed pages and canonicalised duplicates is not a signal — it is noise, and it teaches Google to weight your sitemap less.
Three rules. Every URL returns 200. Every URL is the canonical version. Every URL is one you would be happy to see in search results. If a URL fails any of those, it does not belong in the file.
The lastmod field is worth getting right or leaving out. Google has said it uses the value only where a site has proven it accurate, and stamping today’s date on every URL at every deploy is the fastest way to be ignored. If your CMS cannot produce a real modification date, omit the field.
You can assemble or audit a list quickly with our sitemap generator — point it at an existing sitemap to pull the URLs out, prune the list, and regenerate.
A realistic order of work
- Pull the Crawl stats report and note your response-code mix. This is your baseline.
- Crawl your own site with a desktop crawler and export every internal link that returns a 3xx or 4xx. Fix the links at source.
- Identify parameter patterns generating duplicate URLs. Decide for each: block, noindex, or canonicalise.
- Write the robots.txt rules for the block list. Test them in the Search Console robots.txt tester before deploying.
- Rebuild the sitemap so it contains only canonical, indexable, 200-returning URLs. Resubmit it.
- Wait a month, then pull Crawl stats again and compare the response-code mix.
That last step is the one people skip, and it is the only one that tells you whether any of the work mattered. Crawl behaviour changes slowly — give it four weeks before drawing conclusions.
When not to bother
If your site has a few hundred pages, is fully crawled within days of publishing, and shows a clean response-code mix, crawl budget is not your bottleneck. Spending a fortnight on it will produce nothing measurable.
Crawl waste is a problem of scale and of automatically generated URLs. If you have neither, the same fortnight spent on the content itself will do far more for your rankings. Knowing which problem you actually have is most of the skill.