How do I configure robots.txt for AI crawlers?
Short answer: Add explicit Allow rules for GPTBot, ClaudeBot, Google-Extended, and Bytespider in your robots.txt. Many sites accidentally block them with a wildcard User-agent: * Disallow: / rule. Each AI crawler has a different purpose — train vs. live-search vs. user-triggered browse.
Configuring robots.txt for AI crawlers is now a baseline requirement for any business that wants to appear in ChatGPT, Claude, or Gemini answers. The AI crawlers that matter in 2026 are:
- **GPTBot** (OpenAI): training data + ChatGPT Search index. User-agent: `GPTBot`. - **ChatGPT-User** (OpenAI): triggered when a user explicitly clicks "Browse". User-agent: `ChatGPT-User`. Independent from GPTBot. - **ClaudeBot** (Anthropic): training data + Claude live search. User-agent: `ClaudeBot`. There is also `Claude-Web` for browser-triggered fetches. - **Google-Extended** (Google): controls Gemini training separately from Google Search indexing. User-agent: `Google-Extended`. Blocking it does NOT block Google Search. - **Bytespider** (ByteDance): powers TikTok search + Doubao (Chinese market). User-agent: `Bytespider`.
The most common configuration mistake is a blanket `User-agent: *\ Disallow: /` that blocks all AI crawlers. Another is forgetting that `Allow:` rules are case-sensitive and that a single bad Sitemap line can break parsing for everything below.
ShortcutSistem's Omni Audit reads your robots.txt and grades it against the latest OpenAI, Anthropic, and Google documentation.