What is GPTBot?
Short answer: GPTBot is OpenAI's web crawler. It fetches public web pages to keep ChatGPT's training data and search index fresh. Site owners can block, allow, or partially allow GPTBot via robots.txt. If you block GPTBot, ChatGPT cannot cite your content in its generated answers.
GPTBot is the user-agent string that OpenAI uses to crawl public web pages. It powers both ChatGPT's training-data updates and the live search index used by ChatGPT Search.
You control GPTBot's access through your robots.txt file. Three common configurations:
- `User-agent: GPTBot Disallow: /` — fully blocked. ChatGPT cannot see your site. - `User-agent: GPTBot Allow: / / Sitemap: ...` — fully allowed. - `User-agent: GPTBot Allow: /reports/ Disallow: /api/` — partial. You choose what to expose.
A related user-agent, ChatGPT-User, is used when a user explicitly triggers a browse action. It is independent from GPTBot. Most production setups allow both.
ShortcutSistem's Omni Audit checks your robots.txt against the latest OpenAI, Anthropic, and Google documentation and flags misconfigurations.