AI crawler checker: can AI read your site?
Enter a URL. We check your robots.txt against 13 AI crawlers and fetch your homepage as GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot, so you see both kinds of block: the rule and the firewall. Free, no signup.
The crawlers it checks
Search and user crawlers fetch pages when someone asks a question, so blocking one removes you from that engine's answers. Training crawlers collect data for future models.
| Token | Owner | Used for |
|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT search results |
| ChatGPT-User | OpenAI | Pages a ChatGPT user asks about |
| GPTBot | OpenAI | Model training |
| PerplexityBot | Perplexity | Perplexity search results |
| Perplexity-User | Perplexity | Pages a Perplexity user asks about |
| Claude-SearchBot | Anthropic | Claude search |
| Claude-User | Anthropic | Pages a Claude user asks about |
| ClaudeBot | Anthropic | Model training |
| Googlebot | Google Search, AI Overviews and AI Mode | |
| Google-Extended | Gemini training and grounding (a robots.txt token) | |
| Bingbot | Microsoft | Bing, which feeds Copilot |
| Applebot-Extended | Apple | Apple Intelligence training |
| meta-externalagent | Meta | Meta AI training |
How robots.txt applies to a crawler
- A crawler obeys the group that names it. Only when none does, it follows the
*group. - So a crawler given its own group stops reading your
*rules. Repeat any private paths there. - Among the rules that match a path, the longest one wins. On a tie, Allow wins.
- No robots.txt, or one that returns an HTML page, means everything is allowed.
Per-engine robots.txt lines: ChatGPT, Perplexity, Gemini, Google AI Mode.
Questions
How do I check if AI crawlers can access my site?
Enter your URL above. We read your robots.txt the way the crawlers do and apply it to 13 AI crawlers, then fetch your homepage as a browser and as GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot to see whether a firewall treats them differently.
Why check the firewall as well as robots.txt?
robots.txt is a request; a firewall is a wall. Cloudflare's AI crawler blocking, Bot Fight Mode and Vercel's attack challenge can turn a crawler away even when robots.txt allows it. The checker compares what each crawler got with what a browser got.
Which crawler matters for ChatGPT search?
OAI-SearchBot. OpenAI says sites that block it will not be shown in ChatGPT search answers. GPTBot is the training crawler and can be set separately.
Should I block GPTBot and ClaudeBot?
That is a training choice, not a search one. Blocking them keeps your content out of future models, which can also mean those models know less about your product. Blocking the search crawlers removes you from answers now.
Does blocking Google-Extended affect Google rankings?
Google says it does not affect inclusion or ranking in Google Search. It controls Gemini training and grounding in Gemini Apps.
Why might a result say blocked when my logs show the bot?
We fetch with each crawler's user agent from our own servers. Some firewalls block a crawler name arriving from an unexpected address and allow the real one. Confirm in your CDN or firewall logs before changing anything.
Does this tool store my site?
The result is kept in memory for ten minutes so a repeat check does not fetch your site again. Nothing else is saved and no account is needed.
Is there a limit?
Ten checks per hour from one connection, so the tool cannot be used to hammer someone's site.
What should I do after the crawlers are allowed?
Find out whether the answers name you. The free scan asks ChatGPT, Claude and Google AI Mode your buyers' questions, shows who they name instead of you, and runs the rest of the technical audit.
Does ChatGPT recommend you, or your competitor?
Paste your URL. See what ChatGPT, Perplexity and Google AI answer when your buyers ask, who they name instead of you, and which pages they read. Then we do the work to get you onto those pages.
Free scan, no signup, no card.