Our AI crawler policy

Policy summary

Most sites treat “AI crawlers” as one switch. They are not one switch. Three different kinds of bot arrive with three different bargains, and a single Disallow aimed at all of them usually costs more visibility than it protects.

We publish this page for two reasons. First, we sell Generative Engine Optimization tooling, so our own crawl posture should be inspectable rather than implied. Second, the reasoning transfers: the table below is a worked example you can adapt, and the closing section explains when copying us would be the wrong call.

The three kinds of AI crawler

• Search and answer indexers crawl a page so an assistant can cite and link it. Blocking these removes you from AI answers.
• User-directed fetchers retrieve one page because a person pasted or asked about that URL. Blocking these breaks the experience for someone who already wanted your page.
• Training crawlers collect corpora used to train future models. Blocking these protects your archive but forfeits the baseline familiarity that makes a model name your brand unprompted.

The distinction matters because operators split them across separate user agents. GPTBot is training; OAI-SearchBot is the search index; ChatGPT-User is a live fetch. Disallowing GPTBot does not remove you from ChatGPT, and allowing it does not by itself get you cited.

Three kinds cover the bargains worth reasoning about, but they are not a complete inventory of the agents in our file. OpenAI also ships OAI-AdsBot, which is none of the three; it is disclosed under advertising crawlers below rather than filed under a heading that would misdescribe it.

Search and answer engines — all allowed

These crawlers build the indexes that answer engines read at query time. Every one is allowed on every public route.

Advertising crawlers — allowed

These crawl for advertising purposes rather than to build a query-time answer index, so they are listed separately from the search and answer engines above. We allow them on public routes; allowing one is not a condition of being cited in an answer.

User-directed fetchers — all allowed

These fire when a person asks an assistant about a specific page. The visitor has already chosen us; blocking the fetch would only degrade their answer.

Model training crawlers — allowed

This is the directive most sites get wrong in our direction, and the one most worth explaining. We allow training crawlers because the trade is different for a free-tools brand than for a publisher.

A publisher funded by pageviews loses revenue every time a model answers instead of linking, so ai-train=no defends the business. We have no paywall and no ad inventory. Our growth channel is being named when someone asks an assistant to recommend an SEO tool, and that recognition is built in the training corpus, not at query time. The content we would be protecting is a set of free browser tools we want copied and cited.

Two rows above do not follow from that reasoning, and we would rather name them than let this table imply more coherence than the live file has:

• Applebot-Extended is blocked while every other training crawler is allowed, which contradicts our own ai-train=yes signal. This is not a stance on Apple Intelligence.
• Bytespider is allowed, which follows from the blanket training allowance rather than from a judgement that ByteDance’s crawler earns the same access as OpenAI’s or Anthropic’s. A reasonable policy would block it as a low-value scraper.

Both are inherited from the shared crawler configuration this site uses across a 15-property portfolio, so changing either affects more than this domain. Both are under review. The authoritative directives are always the live file at /robots.txt.

What we block and rate limit

Allowing AI crawlers is not the same as allowing everything. We block extractors that republish or resell content without attribution, because they take the copy and return neither traffic nor citation. SEO index crawlers are allowed but held to a ten-second crawl delay.

The Content-Signal we publish

Access directives say who may fetch a page. Content-Signal says what may be done with it afterwards, across three independent values:

User-Agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /

• search=yes — index this page and link to it.
• ai-input=yes — quote this page in a generated answer.
• ai-train=yes — use this page to train a model.

These are declarations of intent, not access control. They are honoured voluntarily, so anything that genuinely must not be fetched belongs behind authentication rather than a Disallow line.

Copy this policy — or don’t

Copy it if your content is a free utility, documentation, or a product you want recommended, and you have no paywall to defend. Being in the corpus is how assistants learn you exist.

Do not copy it if your revenue depends on pageviews, subscriptions, or a licensed archive. Keep ai-train=no and allow only the search and user-directed agents. You stay quotable and linkable in AI answers without contributing your archive to a training set. That posture — search yes, training no — is the right default for most publishers, and it is what the rest of our portfolio runs.

Build and validate your own directives with the robots.txt and meta robots manager, then confirm what a crawler actually sees with the SEO analyzer.

Frequently asked questions