Google Search Console Crawl Stats Report: Diagnose Crawl Budget

The Crawl Stats report in Google Search Console (Settings > Crawl stats) provides a 90-day server-side view of how Google's crawlers interact with your root-level property (available for Domain properties such as sc-domain:example.com and root-level URL-prefix properties such as https://example.com/, but not subpath URL-prefix properties such as https://example.com/catalog/). While the Page Indexing report tells you which URLs made it into the index, the Crawl Stats report reveals what Googlebot actually requested, how fast your origin responded, how many bytes were transferred, and whether infrastructure errors forced Google to throttle crawling.

Who needs to worry about crawl budget?#

Google defines crawl budget as the combination of crawl capacity limit (how many parallel connections and requests Googlebot can make without overloading your server) and crawl demand (how much Google wants to crawl your URLs based on popularity and staleness). In Google's Large Site Owner's Guide to Managing Crawl Budget, Google states that most sites with fewer than a few thousand URLs do not need to worry about crawl budget at all.

You should actively audit Crawl Stats if your site matches any of these three profiles:

1. Large inventory sites (1 million+ unique URLs) whose content changes at least once a week.
2. Medium-to-large sites (10,000+ unique URLs) with daily content updates, such as news publishers, job boards, or e-commerce catalogs with rapid stock turnover.
3. Any site with a large share of URLs classified as Discovered - currently not indexed in the Page Indexing report, or any site generating faceted filter URLs, infinite calendars, or session parameters.

The three top-level 90-day charts#

At the top of the Crawl Stats report, Search Console plots three daily time series over the trailing 90 days:

Note that average response time in Crawl Stats measures server fetch latency, not Core Web Vitals user experience metrics like Largest Contentful Paint (LCP) or Interaction to Next Paint (INP). However, a slow origin server degrades both crawler capacity and user TTFB.

Host status: robots.txt fetch, DNS resolution, and server connectivity#

When you manage a Domain property (sc-domain:example.com), the Host status section breaks down health per subdomain (www.example.com, shop.example.com, cdn.example.com). Google flags three infrastructure failure modes:

• robots.txt fetch: If your /robots.txt endpoint returns a 5xx server error or times out continuously, Google pauses all crawling of that host for up to 12 hours and falls back to a cached copy for up to 30 days. Test your rules with our robots.txt generator and read our robots.txt guide for status-code handling.
• DNS resolution: Indicates that Google's resolvers encountered SERVFAIL, NXDOMAIN, or timeouts querying your authoritative nameservers.
• Server connectivity: Indicates TCP connection resets, TLS handshake failures, firewall blocks on Googlebot IP ranges, or origin overload.

If Google detects an elevated fail rate in any of those three checks during the past week, it marks the host as having had problems in the past 90 days and displays the exact date window on the failure timeline.

Reading the four Crawl Requests breakdowns#

Below the summary charts, Search Console categorizes every Googlebot request along four dimensions. Clicking any row opens a panel of up to 20 sample URLs with timestamps.

1. By response code#

A healthy production property should concentrate the vast majority of its crawl requests in 200 (OK) and 304 (Not Modified).

• 304 (Not Modified): When your server returns ETag or Last-Modified headers and honors If-Modified-Since / If-None-Match conditional requests, Googlebot receives a zero-body 304 response. That saves download bandwidth and allows Google to recalculate crawl capacity toward changed pages.
• 301 and 302 redirects: High shares of redirects mean internal links or XML sitemaps still point to legacy URLs. Update internal <a> tags to point directly to the final canonical URL.
• 404 (Not Found) and 410 (Gone): Per Google's HTTP status-code documentation, Google treats 404 and 410 identically—all 4xx client errors (except 429) drop previously indexed URLs from the index and gradually reduce crawl frequency in the same way, without hurting your site's overall rankings. However, if thousands of 404 requests hit auto-generated broken paths, trace and remove the internal template link generating them.
• Server error (5xx) and timeout: This is the most dangerous bucket in Crawl Stats. Persistent 500, 502, 503, or 504 responses cause Googlebot to immediately reduce its crawl rate for the entire host.

2. By file type#

This table splits requests across HTML, Image, JavaScript, CSS, PDF, and Other.

If JavaScript and CSS consume a disproportionate share of your crawl requests, check whether your frontend build generates new uncached chunk URLs on every deploy that return 404 when Googlebot renders slightly older cached HTML, or whether third-party widgets trigger dozens of subresource requests per page. Never solve high JS/CSS crawl volume by blocking /_next/static/ or /assets/ in robots.txt; Google's Web Rendering Service (WRS) needs those files to render your layout and links.

3. By purpose: Discovery vs Refresh#

Every request in Crawl Stats is tagged as either Discovery (requesting a URL Google has never crawled before) or Refresh (recrawling a known URL).

• Steady-state mature site (illustrative practitioner heuristic): Often concentrates most requests in Refresh (for example, 70%–90% Refresh and 10%–30% Discovery as an illustrative rule of thumb, not an official Google threshold), depending on publishing velocity and sitemap change frequency.
• High Discovery share (for example, > 60% as an illustrative heuristic): Per Google's Crawl Stats documentation, a discovery spike is completely normal when you launch a new site section, publish a large batch of new content, or unblock previously disallowed paths. However, if Discovery remains persistently high on a mature site that has not added new content, check for a crawl trap—such as faceted e-commerce filter combinations (?color=blue&size=lg&sort=price_asc), infinite pagination, or calendar widgets generating new URL permutations every day.
• New articles sitting in Discovered - currently not indexed while Refresh dominates: Check whether XML sitemaps have accurate <lastmod> timestamps and whether new pages receive internal links from frequently crawled hubs. Read our XML sitemap best practices guide for how Google uses <lastmod> to schedule crawls.

4. By Googlebot type#

This breakdown shows which user-agent token made the requests:

• Smartphone (primary mobile-first indexing crawler)
• Desktop
• Image, Video
• Page resource load (headless Chromium WRS fetching CSS, JS, and XHR endpoints needed to render a page)
• AdsBot, StoreBot

If Page resource load spikes or shows a high failure rate, inspect the rendered DOM in Search Console's URL Inspection tool to see which scripts or API calls are failing during rendering.

Step-by-step workflow to fix crawl budget waste#

1. Eliminate infinite URL spaces in robots.txt: Block non-canonical sort, filter, internal search, and session parameters before Googlebot requests them. Remember that canonical tags and noindex meta tags only work after Googlebot spends crawl budget fetching the page; see our canonical tag guide and test canonical markup with our canonical tag manager.
2. Audit XML sitemaps for 100% 200 OK canonical purity: Remove any 3xx, 404, or noindex URLs from your sitemaps and organize large catalogs with our sitemap index generator.
3. Enable HTTP 304 Not Modified caching: Configure ETag and Last-Modified headers on static and semi-static routes so repeat Refresh crawls finish in milliseconds with zero body transfer.
4. Respond with 429 or 503 (with Retry-After) only during genuine maintenance: Never leave a staging firewall returning 500 or 503 to Googlebot for days, or your production crawl capacity will collapse. Combine Crawl Stats findings with our full technical SEO audit guide.

Frequently asked questions#

Why did Google retire the manual Crawl Rate limiter in Search Console?#

Google retired the legacy manual crawl-rate slider in Search Console on January 8, 2024 after improving Googlebot's automated load-sensing logic. Today, if Googlebot overwhelms your server immediately after a migration or outage, the documented methods to slow it down are returning HTTP 503 or 429 responses on overloaded endpoints or filing a temporary crawl-rate reduction request via Google's Search Console Googlebot report form.

Does blocking a URL in robots.txt remove it from Google's index?#

No. Disallowing a path in robots.txt stops Googlebot from crawling the URL, which saves crawl budget, but if external or internal links point to that URL, Google can still index the URL without a text snippet. To drop an already-indexed URL from search results, allow crawling temporarily and serve a noindex meta tag or HTTP 404/410 status code.

How does Crawl Stats differ from raw server log analysis?#

Per Google's documentation, the Search Console Crawl Stats report covers most Googlebot crawl requests across Google properties over the last 90 days (including subresource loads) without requiring log storage infrastructure, though it is non-exhaustive, may omit some crawl requests, and shows only a limited sample of URLs per category. Raw server access logs record every single request line—including Bingbot, GPTBot, ClaudeBot, and OAI-SearchBot—allowing URL-by-URL join analysis against your site crawl.

Sources#

• Search Console Help: Crawl Stats report documentation
• Google Search Central: Large site owner's guide to managing your crawl budget
• Google Search Central: Reduce the Googlebot crawl rate