Robots.txt Guide: Syntax, Rules, and AI Crawlers

A robots.txt file tells well-behaved crawlers which parts of a site they may request. The format is standardized in RFC 9309, the Robots Exclusion Protocol, and Google documents how its own crawlers read it in How Google interprets the robots.txt specification. This guide covers the parts that cause real-world mistakes: where the file lives, how rules match, what happens when the file fails to load, and how to address AI crawlers by their published tokens.

What robots.txt does and does not do#

Robots.txt manages crawler traffic. Google describes it as a way "to avoid overloading your site with requests" and states plainly that it "is not a mechanism for keeping a web page out of Google" (Google robots.txt introduction). A disallowed URL can still be indexed, without its content, if other pages link to it.

Two consequences follow:

• To keep a page out of search results, let it be crawled and serve a noindex robots meta tag or X-Robots-Tag header, or put it behind authentication. A crawler cannot see a noindex on a page it is not allowed to fetch.
• Robots.txt is not access control. RFC 9309 says the rules are not a form of authorization, and the file itself is public, so listing a sensitive path advertises it.

Where the file must live#

The file is named robots.txt in lowercase and sits at the top level of the host: https://www.example.com/robots.txt. RFC 9309 requires UTF-8 text served as text/plain.

Scope is narrower than many site owners expect. Google applies a robots.txt file only to the protocol, host, and port it is served from. https://example.com, https://www.example.com, and https://shop.example.com each need their own file, and a copy placed in a subdirectory is never checked.

Groups, user agents, and rules#

A file is made of groups. Each group starts with one or more User-agent lines followed by Allow and Disallow rules:

# Applies to every crawler without a more specific group
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help

# Applies only to Googlebot
User-agent: Googlebot
Disallow: /internal-preview/

Sitemap: https://www.example.com/sitemap.xml

How crawlers pick a group:

• User-agent matching is case-insensitive.
• A crawler obeys the most specific group that names it and ignores the rest. In the example above, Googlebot follows only its own group, so it is not blocked from /cart/. Google does not combine a specific group with the * group, so repeat any shared rules inside each named group.
• If several groups name the same crawler, their rules are merged.
• An empty Disallow: line, or a group with no rules, allows everything.

Rule paths are case-sensitive and match from the start of the URL path. Disallow: /search blocks /search, /search/, and /search-results, but not /Search.

Sitemap is not part of RFC 9309, which treats it as an optional extra record. Google supports it, and it can appear anywhere in the file because it is not tied to a group. Use an absolute URL.

How conflicting rules are resolved#

When more than one rule matches a URL, the longest matching path wins. RFC 9309 defines the most specific match as the one with the most octets, and if an Allow and a Disallow rule are equally specific, Allow is used. Google follows the same logic and applies the least restrictive rule on a tie.

User-agent: *
Disallow: /docs/
Allow: /docs/public/

Here /docs/public/setup.html is allowed because /docs/public/ is the longer match, while /docs/drafts/plan.html stays blocked.

Wildcards and end anchors#

RFC 9309 defines two special characters in rule paths:

• * matches zero or more of any character.
• $ anchors the pattern to the end of the URL.

User-agent: *
# Block any URL containing a session parameter
Disallow: /*?sessionid=
# Block PDF files, but not /guide.pdf-notes/
Disallow: /*.pdf$

Lines beginning with # are comments. If a URL contains a literal * or $ you need to match, percent-encode it (for example, %2A).

Status codes, redirects, and caching#

What a crawler does depends on the HTTP response it gets for robots.txt itself:

The 5xx row matters most. A robots.txt that intermittently returns server errors can pause crawling of an entire host. Never return 401 or 403 for robots.txt to slow crawlers down; Google treats those as "no file" and crawls everything.

Crawlers may cache the file. RFC 9309 says a cached copy should not be used for more than 24 hours unless the file is unreachable, and Google generally caches for up to 24 hours. Google also ignores content past 500 KiB, the same minimum parsing limit RFC 9309 sets.

Fields Google ignores#

Google supports only user-agent, allow, disallow, and sitemap. It does not support crawl-delay; adjust Googlebot's load through server responses and Search Console instead. Some other crawlers do read crawl-delay, so leaving it in a crawler-specific group is harmless. Noindex inside robots.txt is not a supported rule; use the robots meta tag instead.

Controlling AI crawlers#

AI companies publish separate tokens for training, search, and user-triggered fetches, so you can treat each purpose differently. Check each vendor's documentation before relying on a token, because names change:

• OpenAI (OpenAI crawlers): GPTBot collects content that may be used to train models, OAI-SearchBot surfaces sites in ChatGPT search, and ChatGPT-User fetches pages when a user asks. OpenAI notes that robots.txt rules may not apply to user-initiated ChatGPT-User requests.
• Anthropic (Anthropic crawler help article): ClaudeBot collects training data, Claude-SearchBot improves search results, and Claude-User retrieves pages for user questions.
• Google (Google common crawlers): Google-Extended is a control token, not a separate crawler. It governs whether crawled content may be used for Gemini training and grounding, and Google states it does not affect inclusion or ranking in Google Search.

A policy that keeps search visibility while opting out of model training looks like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
Allow: /

User-agent: *
Disallow: /cart/

Because each named group replaces the * group for that crawler, the search bots above are not blocked from /cart/. Add the shared Disallow lines to their group if that matters. Remember that robots.txt is voluntary: compliant crawlers honor it, and others need server-level blocking.

Our robots.txt generator for AI crawlers builds these groups for you, and you can compare our own choices on the AI crawler policy page.

A safe change workflow#

1. Fetch the live file and save a copy before editing.
2. Write the smallest rule that achieves the goal, and test it against real URLs from your logs or crawl, including mixed-case and parameterized variants.
3. Deploy, then confirm the file returns 200 with text/plain on every host and protocol you serve.
4. Watch the robots.txt report in Search Console for fetch errors, and check the page indexing report over the following weeks.

Robots.txt is one stage of a wider crawl review. Our technical SEO audit guide places it alongside redirects, canonicals, and XML sitemaps, our Search Console Crawl Stats guide shows how to verify Googlebot fetch health, our llms.txt guide explains how Markdown discovery indexes differ from robots.txt, and the canonical tag guide explains why blocking duplicates in robots.txt is the wrong way to consolidate them.