GlossaryGlossary
How do I configure robots.txt for AI crawlers?How do I configure robots.txt for AI crawlers?
Add explicit Allow rules for GPTBot, ClaudeBot, Google-Extended, and Bytespider in your robots.txt. Many sites accidentally block them with a wildcard User-agent: * Disallow: / rule. Each AI crawler has a different purpose — train vs. live-search vs. user-triggered browse.Add explicit Allow rules for GPTBot, ClaudeBot, Google-Extended, and Bytespider in your robots.txt. Many sites accidentally block them with a wildcard User-agent: * Disallow: / rule. Each AI crawler has a different purpose — train vs. live-search vs. user-triggered browse.
Glossary / How do I configure robots.txt for AI crawlers?
Configuring robots.txt for AI crawlers is now a baseline requirement for any business that wants to appear in ChatGPT, Claude, or Gemini answers. The AI crawlers that matter in 2026 are:
- **GPTBot** (OpenAI): training data + ChatGPT Search index. User-agent: `GPTBot`. - **ChatGPT-User** (OpenAI): triggered when a user explicitly clicks "Browse". User-agent: `ChatGPT-User`. Independent from GPTBot. - **ClaudeBot** (Anthropic): training data + Claude live search. User-agent: `ClaudeBot`. There is also `Claude-Web` for browser-triggered fetches. - **Google-Extended** (Google): controls Gemini training separately from Google Search indexing. User-agent: `Google-Extended`. Blocking it does NOT block Google Search. - **Bytespider** (ByteDance): powers TikTok search + Doubao (Chinese market). User-agent: `Bytespider`.
The most common configuration mistake is a blanket `User-agent: *\ Disallow: /` that blocks all AI crawlers. Another is forgetting that `Allow:` rules are case-sensitive and that a single bad Sitemap line can break parsing for everything below.
ShortcutSistem's Omni Audit reads your robots.txt and grades it against the latest OpenAI, Anthropic, and Google documentation.
Related
| Term | |
|---|---|
| What is GPTBot?GPTBot is OpenAI's web crawler. It fetches public web pages to keep ChatGPT's training data and search index fresh. Site owners can block, allow, or partially allow GPTBot via robots.txt. If you block GPTBot, ChatGPT cannot cite your content in its generated answers. | Read → |
| What is AI Visibility?AI Visibility is how easily ChatGPT, Claude, and Gemini can find, read, and recommend your website. It is the new SEO for the generative-search era. It depends on crawler access, structured data, entity consistency, and original content. | Read → |
| How do LLMs decide which sources to cite?LLMs cite sources that are (1) accessible to their crawlers, (2) structured with schema.org, (3) consistent across the open web as an entity, and (4) original — they contain data or claims not found elsewhere. Backlinks and domain authority help but matter less than primary data. | Read → |
By Filbert Then · Last updated · Suggest a correction
Check your AI readinessCheck your AI readiness
Free. Structural signals only. About 30 seconds.Gratis. Hanya sinyal struktural. Sekitar 30 detik.