DataData
AI crawler list 2026 (updated)AI crawler list 2026 (updated)
Complete list of all known AI bot user agents as of 2026, including OAI-SearchBot, Claude-SearchBot, and Googlebot-Extended.Complete list of all known AI bot user agents as of 2026, including OAI-SearchBot, Claude-SearchBot, and Googlebot-Extended.
Last updated: July 26, 2026 · 11 first-tier crawlers
What changed in 2026
Three new AI crawlers were added to the first-tier list. If you last updated robots.txt in 2025 or earlier, add these immediately.
| New crawler | Operator / UA | Why it matters |
|---|---|---|
| OAI-SearchBot | OpenAI · OAI-SearchBot/1.0 | Powers ChatGPT Search. Separate from GPTBot and ChatGPT-User. |
| Claude-SearchBot | Anthropic · Claude-SearchBot/1.0 | Used when Claude performs real-time web searches during conversations. |
| Googlebot-Extended | Google · Googlebot-Extended | Controls inclusion in Google AI Overviews. Separate from Google-Extended (Gemini training). |
Summary: AI crawlers in 2026
As of 2026, there are 11 first-tier AI web crawlers that websites should allow for maximum AI visibility. The three newest additions are OAI-SearchBot (powers ChatGPT Search), Claude-SearchBot (powers Claude web search), and Googlebot-Extended (controls Google AI Overviews). Combined with the existing GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, DuckAssistBot, Applebot-Extended, and Meta-ExternalAgent, these 11 crawlers cover virtually all AI-generated answers across every major platform. All respect robots.txt. Websites that do not explicitly allow these crawlers risk being invisible to AI search.
All 11 first-tier AI crawlers (2026)
| Name | Operator | User-Agent | Purpose |
|---|---|---|---|
| GPTBot | OpenAI | GPTBot/1.0 | Training data and knowledge base for ChatGPT models |
| ChatGPT-User | OpenAI | ChatGPT-User/1.0 | Real-time browsing when ChatGPT users request web access |
| OAI-SearchBot | OpenAI | OAI-SearchBot/1.0 | Indexes pages for ChatGPT Search feature |
| ClaudeBot | Anthropic | ClaudeBot/1.0 | Training and retrieval data for Claude models |
| Claude-SearchBot | Anthropic | Claude-SearchBot/1.0 | Real-time web search during Claude conversations |
| PerplexityBot | Perplexity AI | PerplexityBot/1.0 | Indexes content for Perplexity search engine answers |
| Google-Extended | Google-Extended | Controls Gemini AI training data usage | |
| Googlebot-Extended | Googlebot-Extended | Controls inclusion in Google AI Overviews | |
| DuckAssistBot | DuckDuckGo | DuckAssistBot/1.0 | Powers DuckDuckGo AI Chat and DuckAssist answers |
| Applebot-Extended | Apple | Applebot-Extended | Apple Intelligence features including Siri and Safari AI |
| Meta-ExternalAgent | Meta | Meta-ExternalAgent/1.0 | Meta AI across WhatsApp, Instagram, and Facebook |
robots.txt template (2026)
Copy this updated template to allow all 11 first-tier AI crawlers.
# robots.txt - AI Crawler Configuration (2026 Update) # Updated to include all first-tier AI crawlers as of 2026 User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: Googlebot-Extended Allow: / User-agent: DuckAssistBot Allow: / User-agent: Applebot-Extended Allow: / User-agent: Meta-ExternalAgent Allow: / # Block sensitive paths User-agent: * Disallow: /admin Disallow: /api Disallow: /private Sitemap: https://yourdomain.com/sitemap.xml
Frequently asked questions
What new AI crawlers were added in 2026?
Three new AI crawlers became widely recognized in 2026: OAI-SearchBot (OpenAI, powers ChatGPT Search), Claude-SearchBot (Anthropic, real-time web search for Claude), and Googlebot-Extended (Google, controls inclusion in AI Overviews). All three respect robots.txt and should be explicitly allowed for maximum AI visibility.
How many AI crawlers exist in 2026?
As of 2026, there are 14 known AI web crawlers operated by 9 companies. The 11 first-tier crawlers that matter most for AI visibility are: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Googlebot-Extended, DuckAssistBot, Applebot-Extended, and Meta-ExternalAgent.
Do I need to update my robots.txt for 2026?
Yes, if your robots.txt was last updated before 2026. You need to add explicit Allow rules for OAI-SearchBot, Claude-SearchBot, and Googlebot-Extended. Without these rules, your site may be invisible to ChatGPT Search, Claude web search, and Google AI Overviews respectively.
What is the difference between Google-Extended and Googlebot-Extended?
Google-Extended controls whether Google uses your content for Gemini model training. Googlebot-Extended (new in late 2025) specifically controls whether your content appears in Google AI Overviews (the AI answer boxes at the top of search results). Blocking Googlebot-Extended means you will not appear in AI Overviews even if you rank well in traditional search.
Is the 2026 AI crawler list different from the 2025 list?
Yes. The 2026 list adds three new crawlers that did not exist or were not widely documented in 2025: OAI-SearchBot, Claude-SearchBot, and Googlebot-Extended. Additionally, Meta-ExternalAgent and DuckAssistBot have significantly increased their crawling activity in 2026.
For the full database with detailed crawler profiles, see the complete AI crawler database.
Is your robots.txt updated for 2026?Is your robots.txt updated for 2026?
Run a free Omni Audit to check if all 11 AI crawlers can access your site. Takes 30 seconds, no signup required.Run a free Omni Audit to check if all 11 AI crawlers can access your site. Takes 30 seconds, no signup required.