Skip to content

Menu

AI Crawler Readiness AI can’t get in. Nobody knew.

Your site may be turning ChatGPT away. We find the block and fix it.Your site may be turning ChatGPT, Perplexity and Claude away. We find the block and fix it, with your yes.

This form needs JavaScript.

Reach us another way:

WhatsApphello@supertoasty.com

Our team will review your details and contact you about the next steps.

  • AI search bots can read prices and hours
  • Changes in your accounts, with your yes

Integration engineer

What AI gets

Built with JavaScript? AI may get a blank.

Many sites load their words with JavaScript. In a 2024 test, no major AI crawler ran it.

What a customer seesCustomer sees

What an AI reader getsAI reader gets

  • Prices. AI has no number to give a buyer.

  • Phone and hours. AI can’t say how to reach you.

  • Services. AI lists only the services it could read.

Vercel and MERJ test, Dec 2024. Gemini was the exception. Source checked 17 Nov 2026.

Where the block hides

Three gates. Most checks test one.

A bot must pass all three. A free robots.txt checker tests only the first.

A common case: robots.txt says yes, the firewall says no.

  1. robots.txt

    Green tick: lets the bot in.

    Asks bots in or out. Cloudflare calls its rules “voluntary”.

  2. Firewall

    Red cross: turns the bot away.

    Decides. Your host, a security plugin or Cloudflare’s Bot Fight Mode can refuse a bot robots.txt welcomes.

  3. Page

    Amber, half empty: the bot gets part of the page.

    Prices and phone numbers must load before any script runs.

Unsure AI reads you at all? See the AI Visibility Audit.

Cloudflare docs, AI Crawler and Agent access policies, updated 2 Nov 2026. Perplexity Crawlers page. Both checked 17 Nov 2026.

Search or training

Keep search bots in. Training is your call.

Search bots put you in AI answers. Training bots copy pages to train models. You can refuse one and keep the other.

The Cloudflare trap. Since 15 Sep 2026, choosing “Block” on Cloudflare’s AI page also turns away Googlebot, “search included”. Its advice: use “Disallow AI Training”.Since 15 Sep 2026, Cloudflare’s “Block” stops Googlebot too. Use “Disallow AI Training”.

AI bots by job, and the yes or no we suggest
BotWhoseWhat it doesOur default
Let in
OAI-SearchBotOpenAIFinds pages for ChatGPT searchYes
ChatGPT-UserOpenAIOpens a page one person asked aboutYes
Claude-SearchBotAnthropicFinds pages for Claude’s answersYes
PerplexityBotPerplexityFinds pages for Perplexity’s answersYes
GooglebotGoogleGoogle Search and its AI answersYes
BingbotMicrosoftBing searchYes
ApplebotAppleSpotlight, Siri and Safari searchYes
Your call
GPTBotOpenAICopies pages to train modelsYour call
ClaudeBotAnthropicCopies pages to train ClaudeYour call
Google-ExtendedGoogleGemini training and Gemini app answers, not SearchYour call

Cloudflare blog, 15 Sep 2026. Bot rows from the OpenAI, Anthropic, Perplexity, Google and Apple crawler pages. All checked 17 Nov 2026.

Chose “Block” on Cloudflare’s AI page? Check Google still gets in.

Get Started

What you get

Five things you keep. Your log proves it.

Your server log shows each bot’s visit and your site’s answer, before and after the fix.

  • A bot table. Each AI bot, its job, and your yes or no.
  • A three-gate result. A tick or cross per gate, with the log line.
  • A no-script check. What your key pages lose without scripts.
  • The fixes. Made in your accounts, each with your yes.
  • A settings note. What changed, where, and how to undo it.

200 here’s the page. 403 not allowed.

Straight answers

No AI file for Google. Its guide says so.

Some agencies add an llms.txt file to the fix. Google says its AI answers don’t need one.

llms.txt

“You don’t need to create new machine readable files, AI text files, or markup to appear in these features.”

Google Search Central, AI features and your website
  • No promise you’ll be named. Being read comes first.

  • No change without your yes. You see what each change touches first.

  • No training call made for you. That one is yours.

Your team

An engineer finds it. A second checks it.

Your robots.txt, firewall and pages are tested by an integration engineer, then a quality specialist checks the fix.

Integration engineer.

Writes down how everything fits together.

After you press Submit

  1. 1We review your details.
  2. 2We talk it through with you.
  3. 3We propose a solution.
  • The systems, the code and the accounts are yours.
  • Priority access to Luna. Partners go to the front of the queue with our AI assistant.
  • We cover operations, visibility, websites and growth with one team, so the work is planned together.
  • We accept only a limited number of partnerships each quarter.

Questions

Your questions, their own answers.

Answers from OpenAI, Google, Cloudflare and Perplexity’s own pages.

What is AI crawler readiness?

AI Crawler Readiness is SuperToasty’s work to let the bots behind ChatGPT, Claude, Perplexity and Google’s AI answers reach and read a business’s website. SuperToasty checks robots.txt, the firewall or host, and what each page shows without scripts, then fixes what blocks them, with the owner’s yes.

A question we missed? Ask it in the form’s last box.

Get Started
Should I block AI crawlers on my website?

Blocking AI crawlers covers three separate decisions. Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) decide whether your pages can be shown and linked in AI answers. Training bots (GPTBot, ClaudeBot) copy pages to teach future models. Live-visit bots open one page because one person asked. A business that wants enquiries usually lets search and live visits in. Training is the owner’s call.

Why can’t ChatGPT open my website?

ChatGPT can fail to open a website for three common reasons. robots.txt may disallow OAI-SearchBot or ChatGPT-User. A firewall, Cloudflare setting or security plugin may refuse OpenAI’s visits, which robots.txt never shows. Or the page builds its words with scripts the bot does not run. OpenAI advises allowing OAI-SearchBot in robots.txt and “allowing requests from our published IP ranges”.

Does blocking AI bots on Cloudflare block Google too?

Blocking AI training on Cloudflare can block Googlebot. Since 15 September 2026, Cloudflare’s “Block” choice for training also stops Googlebot, Bingbot and Applebot, “search included”, because each crawls for search and training under one name. To stop training and keep search, Cloudflare says to use “Disallow AI Training”. Bot Fight Mode and custom firewall rules are separate switches to check.

Do AI crawlers respect robots.txt?

AI crawlers mostly respect robots.txt when they crawl on their own. Anthropic says its bots honour “industry standard directives in robots.txt”, and OpenAI and Perplexity both use robots.txt to manage their search bots. Live-visit bots differ: OpenAI says for ChatGPT-User “robots.txt rules may not apply”, and Perplexity says its user fetcher “generally ignores robots.txt rules”. Only a firewall rule actually enforces a no.

How do I know a visit is really OAI-SearchBot or Googlebot?

A visit is really OAI-SearchBot or Googlebot when it comes from the IP addresses OpenAI or Google publish for that bot. The name alone can be faked: Google warns “the HTTP user agent string can be spoofed”. Perplexity says to “match each User-Agent against its own IP ranges”. A firewall rule can check visits against those lists.

Do AI crawlers read pages built with JavaScript?

Most AI crawlers did not run JavaScript when Vercel and MERJ tested them in December 2024: “none of the major AI crawlers currently render JavaScript”, covering OpenAI’s, Anthropic’s and Perplexity’s bots. Gemini was the exception, using Google’s own rendering. Bots change, so we test your pages as they are today.

Is Cloudflare’s AI Crawl Control free?

Cloudflare’s AI Crawl Control is “Available on all plans”, according to Cloudflare’s docs (updated 14 August 2026). It shows which AI bots visit your site and lets you allow or block each one. The bots themselves charge nothing to visit. The work is finding which of the three gates says no, then proving the fix in your server log.

How long until ChatGPT and other AI tools see the change?

A robots.txt change takes about a day to reach the AI companies. OpenAI says “it can take ~24 hours” for ChatGPT search to adjust, and Perplexity says “up to 24 hours”. A firewall change works on the bot’s next visit, and the server log shows it. Being read is only the start. Whether an assistant names you depends on what your pages say.

Do I need this, or an AI visibility audit?

AI crawler readiness suits a business that suspects something turns AI bots away: a site behind Cloudflare or a security plugin, or one that blocked “AI” last year. The AI Visibility Audit suits an owner who does not yet know whether AI names them at all. The audit looks across every cause once; crawler readiness finds the exact block and fixes it. A site Google cannot find needs ordinary SEO first.

Answers quote OpenAI, Anthropic, Perplexity, Google, Cloudflare and Vercel pages, all checked 17 Nov 2026.

A question we missed? Ask it in the form’s last box.

Get Started

Keep going

Not quite what you need?

See the rest of this group, or the service closest to this one.