GlossaryGlossary
What is GPTBot?What is GPTBot?
GPTBot is OpenAI's web crawler. It fetches public web pages to keep ChatGPT's training data and search index fresh. Site owners can block, allow, or partially allow GPTBot via robots.txt. If you block GPTBot, ChatGPT cannot cite your content in its generated answers.GPTBot is OpenAI's web crawler. It fetches public web pages to keep ChatGPT's training data and search index fresh. Site owners can block, allow, or partially allow GPTBot via robots.txt. If you block GPTBot, ChatGPT cannot cite your content in its generated answers.
Glossary / What is GPTBot?
GPTBot is the user-agent string that OpenAI uses to crawl public web pages. It powers both ChatGPT's training-data updates and the live search index used by ChatGPT Search.
You control GPTBot's access through your robots.txt file. Three common configurations:
- `User-agent: GPTBot Disallow: /` — fully blocked. ChatGPT cannot see your site. - `User-agent: GPTBot Allow: / / Sitemap: ...` — fully allowed. - `User-agent: GPTBot Allow: /reports/ Disallow: /api/` — partial. You choose what to expose.
A related user-agent, ChatGPT-User, is used when a user explicitly triggers a browse action. It is independent from GPTBot. Most production setups allow both.
ShortcutSistem's Omni Audit checks your robots.txt against the latest OpenAI, Anthropic, and Google documentation and flags misconfigurations.
Sources
Related
| Term | |
|---|---|
| What is AI Visibility?AI Visibility is how easily ChatGPT, Claude, and Gemini can find, read, and recommend your website. It is the new SEO for the generative-search era. It depends on crawler access, structured data, entity consistency, and original content. | Read → |
By Filbert Then · Last updated · Suggest a correction
Check your AI readinessCheck your AI readiness
Free. Structural signals only. About 30 seconds.Gratis. Hanya sinyal struktural. Sekitar 30 detik.