← All tools

Free tool

AI crawler checker for robots.txt

Paste your site's robots.txt and see, crawler by crawler, whether ChatGPT, Claude, Perplexity, Gemini, Copilot and the other AI assistants can read your pages.

  • Many sites block AI crawlers without knowing it: a security plugin or a copied robots.txt template is enough.
  • Blocking training crawlers is a legitimate choice. Blocking search crawlers takes you out of answers that cite their sources.

Paste a robots.txt to see the result.

    Quick guide

    How to use it

    1. 1

      Type your site address and open its robots.txt, or open yoursite.com/robots.txt yourself.

    2. 2

      Copy all of the text and paste it into the box: the check runs straight away.

    3. 3

      Read the table and, if needed, copy the suggested lines to let the search crawlers back in.

    FAQ

    Frequently asked questions

    What if my site has no robots.txt?

    If yoursite.com/robots.txt returns a 404 error, every crawler is allowed in. That is not a problem, but it is worth having one that points to your sitemap.

    What is the difference between search and training crawlers?

    Search crawlers, such as OAI-SearchBot or PerplexityBot, read pages to answer questions and cite sources. Training crawlers, such as GPTBot or CCBot, collect text to train models. You can let the first in and keep the second out.

    Is my robots.txt sent anywhere?

    No. The check happens in your browser. The button opens your robots.txt in a new tab, but this page does not download it or send it to anyone.

    Want a website that does all this for you?

    We'll prepare a free demo, already optimized for Google and AI.

    Request the free demo →

    Free demo

    Let's see what your site could look like.

    Write to us on WhatsApp or by email: we answer within 24 working hours with first ideas for your site.

    Message us on WhatsAppEmail us

    info@bianchiweb.com+39 352 074 7399