AI Crawler Readiness AI can’t get in. Nobody knew.
Your site may be turning ChatGPT away. We find the block and fix it.Your site may be turning ChatGPT, Perplexity and Claude away. We find the block and fix it, with your yes.
- AI search bots can read prices and hours
- Changes in your accounts, with your yes
What AI gets
Built with JavaScript? AI may get a blank.
Many sites load their words with JavaScript. In a 2024 test, no major AI crawler ran it.
What a customer seesCustomer sees
What an AI reader getsAI reader gets
Prices. AI has no number to give a buyer.
Phone and hours. AI can’t say how to reach you.
Services. AI lists only the services it could read.
Vercel and MERJ test, Dec 2024. Gemini was the exception. Source checked 17 Nov 2026.
Where the block hides
Three gates. Most checks test one.
A bot must pass all three. A free robots.txt checker tests only the first.
A common case: robots.txt says yes, the firewall says no.
-
robots.txt
Green tick: lets the bot in.Asks bots in or out. Cloudflare calls its rules “voluntary”.
-
Firewall
Red cross: turns the bot away.Decides. Your host, a security plugin or Cloudflare’s Bot Fight Mode can refuse a bot robots.txt welcomes.
-
Page
Amber, half empty: the bot gets part of the page.Prices and phone numbers must load before any script runs.
Unsure AI reads you at all? See the AI Visibility Audit.
Cloudflare docs, AI Crawler and Agent access policies, updated 2 Nov 2026. Perplexity Crawlers page. Both checked 17 Nov 2026.
Search or training
Keep search bots in. Training is your call.
Search bots put you in AI answers. Training bots copy pages to train models. You can refuse one and keep the other.
Block
or
Disallow AI Training
The Cloudflare trap. Since 15 Sep 2026, choosing “Block” on Cloudflare’s AI page also turns away Googlebot, “search included”. Its advice: use “Disallow AI Training”.Since 15 Sep 2026, Cloudflare’s “Block” stops Googlebot too. Use “Disallow AI Training”.
| Bot | Whose | What it does | Our default |
|---|---|---|---|
| Let in | |||
| OAI-SearchBot | OpenAI | Finds pages for ChatGPT search | Yes |
| ChatGPT-User | OpenAI | Opens a page one person asked about | Yes |
| Claude-SearchBot | Anthropic | Finds pages for Claude’s answers | Yes |
| PerplexityBot | Perplexity | Finds pages for Perplexity’s answers | Yes |
| Googlebot | Google Search and its AI answers | Yes | |
| Bingbot | Microsoft | Bing search | Yes |
| Applebot | Apple | Spotlight, Siri and Safari search | Yes |
| Your call | |||
| GPTBot | OpenAI | Copies pages to train models | Your call |
| ClaudeBot | Anthropic | Copies pages to train Claude | Your call |
| Google-Extended | Gemini training and Gemini app answers, not Search | Your call | |
Cloudflare blog, 15 Sep 2026. Bot rows from the OpenAI, Anthropic, Perplexity, Google and Apple crawler pages. All checked 17 Nov 2026.
Chose “Block” on Cloudflare’s AI page? Check Google still gets in.
Get StartedWhat you get
Five things you keep. Your log proves it.
Your server log shows each bot’s visit and your site’s answer, before and after the fix.
- A bot table. Each AI bot, its job, and your yes or no.
- A three-gate result. A tick or cross per gate, with the log line.
- A no-script check. What your key pages lose without scripts.
- The fixes. Made in your accounts, each with your yes.
- A settings note. What changed, where, and how to undo it.
GET /prices · Googlebot200GET /prices · PerplexityBot403GET /contact · OAI-SearchBot200
GET /prices · PerplexityBot200
200 here’s the page. 403 not allowed.
Straight answers
No AI file for Google. Its guide says so.
Some agencies add an llms.txt file to the fix. Google says its AI answers don’t need one.
“You don’t need to create new machine readable files, AI text files, or markup to appear in these features.”
No promise you’ll be named. Being read comes first.
No change without your yes. You see what each change touches first.
No training call made for you. That one is yours.
Your team
An engineer finds it. A second checks it.
Your robots.txt, firewall and pages are tested by an integration engineer, then a quality specialist checks the fix.

Quality specialist.
Checks the details your customers will see.

Account manager.
Keeps you updated in plain language, without the jargon.
- 1We review your details.
- 2We talk it through with you.
- 3We propose a solution.
- The systems, the code and the accounts are yours.
- Priority access to Luna. Partners go to the front of the queue with our AI assistant.
- We cover operations, visibility, websites and growth with one team, so the work is planned together.
- We accept only a limited number of partnerships each quarter.
Questions
Your questions, their own answers.
Answers from OpenAI, Google, Cloudflare and Perplexity’s own pages.
What is AI crawler readiness?
AI Crawler Readiness is SuperToasty’s work to let the bots behind ChatGPT, Claude, Perplexity and Google’s AI answers reach and read a business’s website. SuperToasty checks robots.txt, the firewall or host, and what each page shows without scripts, then fixes what blocks them, with the owner’s yes.
robots.txt Example
User-agent: OAI-SearchBotAllow: /
User-agent: GPTBotDisallow: /
Should I block AI crawlers on my website?
Blocking AI crawlers covers three separate decisions. Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) decide whether your pages can be shown and linked in AI answers. Training bots (GPTBot, ClaudeBot) copy pages to teach future models. Live-visit bots open one page because one person asked. A business that wants enquiries usually lets search and live visits in. Training is the owner’s call.
Why can’t ChatGPT open my website?
ChatGPT can fail to open a website for three common reasons. robots.txt may disallow OAI-SearchBot or ChatGPT-User. A firewall, Cloudflare setting or security plugin may refuse OpenAI’s visits, which robots.txt never shows. Or the page builds its words with scripts the bot does not run. OpenAI advises allowing OAI-SearchBot in robots.txt and “allowing requests from our published IP ranges”.
Does blocking AI bots on Cloudflare block Google too?
Blocking AI training on Cloudflare can block Googlebot. Since 15 September 2026, Cloudflare’s “Block” choice for training also stops Googlebot, Bingbot and Applebot, “search included”, because each crawls for search and training under one name. To stop training and keep search, Cloudflare says to use “Disallow AI Training”. Bot Fight Mode and custom firewall rules are separate switches to check.
Do AI crawlers respect robots.txt?
AI crawlers mostly respect robots.txt when they crawl on their own. Anthropic says its bots honour “industry standard directives in robots.txt”, and OpenAI and Perplexity both use robots.txt to manage their search bots. Live-visit bots differ: OpenAI says for ChatGPT-User “robots.txt rules may not apply”, and Perplexity says its user fetcher “generally ignores robots.txt rules”. Only a firewall rule actually enforces a no.
How do I know a visit is really OAI-SearchBot or Googlebot?
A visit is really OAI-SearchBot or Googlebot when it comes from the IP addresses OpenAI or Google publish for that bot. The name alone can be faked: Google warns “the HTTP user agent string can be spoofed”. Perplexity says to “match each User-Agent against its own IP ranges”. A firewall rule can check visits against those lists.
Do AI crawlers read pages built with JavaScript?
Most AI crawlers did not run JavaScript when Vercel and MERJ tested them in December 2024: “none of the major AI crawlers currently render JavaScript”, covering OpenAI’s, Anthropic’s and Perplexity’s bots. Gemini was the exception, using Google’s own rendering. Bots change, so we test your pages as they are today.
Is Cloudflare’s AI Crawl Control free?
Cloudflare’s AI Crawl Control is “Available on all plans”, according to Cloudflare’s docs (updated 14 August 2026). It shows which AI bots visit your site and lets you allow or block each one. The bots themselves charge nothing to visit. The work is finding which of the three gates says no, then proving the fix in your server log.
How long until ChatGPT and other AI tools see the change?
A robots.txt change takes about a day to reach the AI companies. OpenAI says “it can take ~24 hours” for ChatGPT search to adjust, and Perplexity says “up to 24 hours”. A firewall change works on the bot’s next visit, and the server log shows it. Being read is only the start. Whether an assistant names you depends on what your pages say.
Do I need this, or an AI visibility audit?
AI crawler readiness suits a business that suspects something turns AI bots away: a site behind Cloudflare or a security plugin, or one that blocked “AI” last year. The AI Visibility Audit suits an owner who does not yet know whether AI names them at all. The audit looks across every cause once; crawler readiness finds the exact block and fixes it. A site Google cannot find needs ordinary SEO first.
Answers quote OpenAI, Anthropic, Perplexity, Google, Cloudflare and Vercel pages, all checked 17 Nov 2026.
Keep going
Not quite what you need?
See the rest of this group, or the service closest to this one.

