DataData
Complete list of AI web crawlers (2026)Complete list of AI web crawlers (2026)
Every known AI web crawler as of July 2026 — user-agent strings, operators, purposes, and a ready-to-use robots.txt template. Updated monthly.Every known AI web crawler as of July 2026 — user-agent strings, operators, purposes, and a ready-to-use robots.txt template. Updated monthly.
Last updated: July 26, 2026 · 14 crawlers tracked
Summary
As of 2026, there are 14 known AI web crawlers operated by 9 companies. The most important ones to allow in your robots.txt are: GPTBot and OAI-SearchBot (OpenAI/ChatGPT), ClaudeBot (Anthropic/Claude), PerplexityBot (Perplexity AI), Google-Extended and Googlebot-Extended (Google Gemini and AI Overviews), Applebot-Extended (Apple Intelligence), and Meta-ExternalAgent (Meta AI). These 8 crawlers cover 95%+ of AI-generated answers worldwide. All respect robots.txt — you can selectively allow or block each one. Blocking all of them makes your website invisible to AI systems. Use the robots.txt template below to allow all major AI crawlers while blocking internal pages.
All AI crawlers
| Crawler | Operator | Type | First seen |
|---|---|---|---|
| GPTBotIndexes web content for ChatGPT training data and knowledgeUA: GPTBot/1.0 · robots.txt: Yes · DocsPrimary OpenAI crawler. Blocking this prevents your content from appearing in ChatGPT responses. | OpenAI | Training + Knowledge | August 2023 |
| ChatGPT-UserReal-time browsing when a ChatGPT user clicks "Browse the web"UA: ChatGPT-User/1.0 · robots.txt: Yes · DocsTriggered by user action. Blocking this prevents ChatGPT from reading your site in real-time conversations. | OpenAI | Real-time browsing | March 2024 |
| OAI-SearchBotPowers ChatGPT Search (the search feature inside ChatGPT)UA: OAI-SearchBot/1.0 · robots.txt: Yes · DocsIndexes pages for ChatGPT's built-in search. Separate from GPTBot training. | OpenAI | Search index | October 2024 |
| ClaudeBotIndexes web content for Claude training and knowledgeUA: ClaudeBot/1.0 · robots.txt: Yes · DocsPrimary Anthropic crawler. Blocking prevents your content from being in Claude's knowledge. | Anthropic | Training + Knowledge | March 2024 |
| Claude-SearchBotReal-time web search for Claude (search-grounded responses)UA: Claude-SearchBot/1.0 · robots.txt: Yes · DocsUsed when Claude searches the web during a conversation. | Anthropic | Real-time search | Early 2025 |
| PerplexityBotIndexes content for Perplexity search engine responsesUA: PerplexityBot/1.0 · robots.txt: Yes · DocsPerplexity is one of the most active AI citers. High-value to allow. | Perplexity AI | Search index + Real-time | Early 2024 |
| Google-ExtendedControls whether Google uses your content for Gemini and AI trainingUA: Google-Extended · robots.txt: Yes · DocsSeparate from Googlebot. Blocking this does NOT affect Google Search ranking — only AI training. | AI training (Gemini, Bard) | September 2023 | |
| Googlebot-ExtendedPost-2025 partition of Google-Extended for AI Overviews specificallyUA: Googlebot-Extended · robots.txt: Yes · DocsControls inclusion in Google AI Overviews (SGE). Blocking means you won't appear in AI answer boxes. | Google AI Overviews | Late 2025 | |
| DuckAssistBotPowers DuckDuckGo AI Chat and DuckAssist featuresUA: DuckAssistBot/1.0 · robots.txt: Yes · DocsPowers the AI chat feature in DuckDuckGo search. | DuckDuckGo | AI-powered search | Mid 2024 |
| Applebot-ExtendedIndexes content for Apple Intelligence features (Siri, Safari AI)UA: Applebot-Extended · robots.txt: Yes · DocsControls whether Apple Intelligence uses your content. Separate from standard Applebot. | Apple | Apple Intelligence | Mid 2024 |
| Meta-ExternalAgentIndexes content for Meta AI (used across Facebook, Instagram, WhatsApp)UA: Meta-ExternalAgent/1.0 · robots.txt: Yes · DocsPowers Meta AI responses across Meta platforms (3.5B+ users). | Meta | AI training + responses | Late 2024 |
| CCBotOpen web crawl dataset used by most AI companies for trainingUA: CCBot/2.0 · robots.txt: Yes · DocsCommon Crawl feeds training data to most open-source and commercial LLMs. Allowing this gives broadest AI visibility. | Common Crawl | Open training data | 2011 |
| BytespiderIndexes content for ByteDance AI products and TikTok searchUA: Bytespider · robots.txt: YesVery aggressive crawler (high volume). Many sites block due to bandwidth consumption. Powers TikTok search and Doubao AI. | ByteDance (TikTok) | AI training + search | 2022 |
| AmazonbotIndexes content for Alexa AI and Amazon searchUA: Amazonbot/0.1 · robots.txt: Yes · DocsPowers Alexa answers and Amazon product search context. | Amazon | AI assistant (Alexa) | 2022 |
Ready-to-use robots.txt template
Copy this template to allow all 11 first-tier AI crawlers. Replace yourdomain.com with your actual domain.
# robots.txt — Allow all major AI crawlers (2026)
# Last updated: July 2026
# Source: https://www.shortcutsistem.com/data/ai-crawler-list
# Default: allow all
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /private/
# ── OpenAI ──
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
# ── Anthropic ──
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# ── Google AI ──
User-agent: Google-Extended
Allow: /
User-agent: Googlebot-Extended
Allow: /
# ── Perplexity ──
User-agent: PerplexityBot
Allow: /
# ── Apple ──
User-agent: Applebot-Extended
Allow: /
# ── Meta ──
User-agent: Meta-ExternalAgent
Allow: /
# ── DuckDuckGo ──
User-agent: DuckAssistBot
Allow: /
# ── Open Data ──
User-agent: CCBot
Allow: /
# ── Optional: Block aggressive crawlers ──
# User-agent: Bytespider
# Disallow: /
Sitemap: https://yourdomain.com/sitemap.xmlFrequently asked questions
What AI crawlers should I allow in robots.txt?
At minimum, allow GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Googlebot-Extended, OAI-SearchBot, Applebot-Extended, and Meta-ExternalAgent. These 8 cover 95%+ of AI-generated answers.
Does blocking GPTBot affect my Google ranking?
No. GPTBot only affects ChatGPT. Your Google Search ranking is determined by Googlebot, not GPTBot. However, blocking Googlebot-Extended will remove you from Google AI Overviews.
How can I check if AI crawlers are blocked?
Use Omni Audit (free, no signup) at shortcutsistem.com/omni-audit — it checks all 11 AI crawlers against your robots.txt and shows exactly which ones are blocked.
What is the difference between GPTBot and ChatGPT-User?
GPTBot crawls for training/knowledge data continuously. ChatGPT-User is triggered only when a user asks ChatGPT to browse a specific URL in real-time.
Should I block Bytespider?
It depends. Bytespider (ByteDance/TikTok) is very aggressive and can consume significant bandwidth. If you don't need TikTok AI visibility, blocking is reasonable. Most sites block it.
Maintained by ShortcutSistem. Last update: July 26, 2026. Found an error? Let us know.
Check if these crawlers can access your siteCheck if these crawlers can access your site
Omni Audit scans your robots.txt against all 11 first-tier AI crawlers in 30 seconds. Free, no signup required.Omni Audit scans your robots.txt against all 11 first-tier AI crawlers in 30 seconds. Free, no signup required.