Complete List of AI Web Crawlers (2026)
Every known AI web crawler as of July 2026 — their user-agent strings, what they do, and how to configure your robots.txt to allow or block them. Updated monthly.
Last updated: July 26, 2026 · 14 crawlers tracked · Check your site's AI visibility free →
Summary
As of 2026, there are 14 known AI web crawlers operated by 9 companies. The most important ones to allow in your robots.txt are: GPTBot and OAI-SearchBot (OpenAI/ChatGPT), ClaudeBot (Anthropic/Claude), PerplexityBot (Perplexity AI), Google-Extended and Googlebot-Extended (Google Gemini and AI Overviews), Applebot-Extended (Apple Intelligence), and Meta-ExternalAgent (Meta AI). These 8 crawlers cover 95%+ of AI-generated answers worldwide. All respect robots.txt — you can selectively allow or block each one. Blocking all of them makes your website invisible to AI systems, meaning AI will never recommend your business when users ask relevant questions. Use the robots.txt template below to allow all major AI crawlers while blocking internal pages.
All AI Crawlers
Click any crawler name for details.
GPTBot/1.0GPTBotTraining + Knowledge
GPTBot/1.0Purpose: Indexes web content for ChatGPT training data and knowledge
Primary OpenAI crawler. Blocking this prevents your content from appearing in ChatGPT responses.
Allow: /
ChatGPT-User/1.0ChatGPT-UserReal-time browsing
ChatGPT-User/1.0Purpose: Real-time browsing when a ChatGPT user clicks "Browse the web"
Triggered by user action. Blocking this prevents ChatGPT from reading your site in real-time conversations.
Allow: /
OAI-SearchBot/1.0OAI-SearchBotSearch index
OAI-SearchBot/1.0Purpose: Powers ChatGPT Search (the search feature inside ChatGPT)
Indexes pages for ChatGPT's built-in search. Separate from GPTBot training.
Allow: /
ClaudeBot/1.0ClaudeBotTraining + Knowledge
ClaudeBot/1.0Purpose: Indexes web content for Claude training and knowledge
Primary Anthropic crawler. Blocking prevents your content from being in Claude's knowledge.
Allow: /
Claude-SearchBot/1.0Claude-SearchBotReal-time search
Claude-SearchBot/1.0Purpose: Real-time web search for Claude (search-grounded responses)
Used when Claude searches the web during a conversation.
Allow: /
PerplexityBot/1.0PerplexityBotSearch index + Real-time
PerplexityBot/1.0Purpose: Indexes content for Perplexity search engine responses
Perplexity is one of the most active AI citers. High-value to allow.
Allow: /
Google-ExtendedGoogle-ExtendedAI training (Gemini, Bard)
Google-ExtendedPurpose: Controls whether Google uses your content for Gemini and AI training
Separate from Googlebot. Blocking this does NOT affect Google Search ranking — only AI training.
Allow: /
Googlebot-ExtendedGooglebot-ExtendedGoogle AI Overviews
Googlebot-ExtendedPurpose: Post-2025 partition of Google-Extended for AI Overviews specifically
Controls inclusion in Google AI Overviews (SGE). Blocking means you won't appear in AI answer boxes.
Allow: /
DuckAssistBot/1.0DuckAssistBotAI-powered search
DuckAssistBot/1.0Purpose: Powers DuckDuckGo AI Chat and DuckAssist features
Powers the AI chat feature in DuckDuckGo search.
Allow: /
Applebot-ExtendedApplebot-ExtendedApple Intelligence
Applebot-ExtendedPurpose: Indexes content for Apple Intelligence features (Siri, Safari AI)
Controls whether Apple Intelligence uses your content. Separate from standard Applebot.
Allow: /
Meta-ExternalAgent/1.0Meta-ExternalAgentAI training + responses
Meta-ExternalAgent/1.0Purpose: Indexes content for Meta AI (used across Facebook, Instagram, WhatsApp)
Powers Meta AI responses across Meta platforms (3.5B+ users).
Allow: /
CCBot/2.0CCBotOpen training data
CCBot/2.0Purpose: Open web crawl dataset used by most AI companies for training
Common Crawl feeds training data to most open-source and commercial LLMs. Allowing this gives broadest AI visibility.
Allow: /
BytespiderBytespiderAI training + search
BytespiderPurpose: Indexes content for ByteDance AI products and TikTok search
Very aggressive crawler (high volume). Many sites block due to bandwidth consumption. Powers TikTok search and Doubao AI.
Allow: /
Amazonbot/0.1AmazonbotAI assistant (Alexa)
Amazonbot/0.1Purpose: Indexes content for Alexa AI and Amazon search
Powers Alexa answers and Amazon product search context.
Allow: /
Ready-to-Use robots.txt Template
Copy this template to allow all 11 first-tier AI crawlers. Replace yourdomain.com with your actual domain.
# robots.txt — Allow all major AI crawlers (2026)
# Last updated: July 2026
# Source: https://www.shortcutsistem.com/data/ai-crawler-list
# Default: allow all
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /private/
# ── OpenAI ──
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
# ── Anthropic ──
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# ── Google AI ──
User-agent: Google-Extended
Allow: /
User-agent: Googlebot-Extended
Allow: /
# ── Perplexity ──
User-agent: PerplexityBot
Allow: /
# ── Apple ──
User-agent: Applebot-Extended
Allow: /
# ── Meta ──
User-agent: Meta-ExternalAgent
Allow: /
# ── DuckDuckGo ──
User-agent: DuckAssistBot
Allow: /
# ── Open Data ──
User-agent: CCBot
Allow: /
# ── Optional: Block aggressive crawlers ──
# User-agent: Bytespider
# Disallow: /
Sitemap: https://yourdomain.com/sitemap.xmlCheck if these crawlers can access YOUR site
Omni Audit scans your robots.txt against all 11 first-tier AI crawlers in 30 seconds. Free, no signup required.
Run Free AI Visibility Audit →Frequently Asked Questions
What AI crawlers should I allow in robots.txt?
Does blocking GPTBot affect my Google ranking?
How can I check if AI crawlers are blocked?
What is the difference between GPTBot and ChatGPT-User?
Should I block Bytespider?
This database is maintained by ShortcutSistem and updated monthly. Last update: July 26, 2026. Found an error or know of a new AI crawler? Let us know.