Crawl Budget: What It Is and How to Optimize It

What is crawl budget?

Crawl budget is the number of URLs a search engine will crawl on your site within a given timeframe. It's determined by two things: crawl capacity (how much load your server can handle without slowing down) and crawl demand (how much Google wants your content, based on popularity and how often it changes).

How does Google decide my crawl budget?

Google balances crawl capacity against crawl demand. Fast, healthy servers earn more crawling; slow responses and errors reduce it. Sites with popular, frequently-updated content generate more demand, so their pages get revisited more often. There's no fixed number — it adjusts continuously.

That means crawl budget isn't something you're assigned once. Every improvement to server speed, content freshness, and site health nudges it upward, and every wave of errors or slow responses pulls it back down.

What wastes crawl budget?

Every wasted fetch is a fetch not spent on the pages you want indexed. The big offenders are:

• Infinite URL spaces from faceted navigation and filter parameters
• Soft-404s (pages that return 200 but have no real content)
• Long redirect chains
• Duplicate pages and session-ID URLs
• Hacked or spammy pages
• Huge numbers of low-value, thin pages

How to optimize or increase crawl budget

Make every crawl request count toward pages that matter:

• Speed up your server — faster responses raise your crawl capacity
• Fix broken links and long redirect chains
• Remove or consolidate duplicate and low-value pages
• Keep your XML sitemap clean and current
• Block genuinely useless URLs (like faceted-filter combinations) in robots.txt

Do small sites need to worry about crawl budget?

Mostly no — this is a common misconception. Google says crawl budget is only a real concern for very large sites (roughly a million+ pages) or sites with rapidly changing content in the tens of thousands of pages. If your site has a few hundred or thousand pages, focus on content quality and crawlability basics instead.

That said, the things that waste crawl budget — broken links, redirect chains, duplicates — hurt sites of every size for other reasons, so fixing them is never wasted work. A site crawl surfaces all of them in one scan.