Free tool · no account

Which AI crawlers does your robots.txt let in?

Enter a domain. We fetch its robots.txt and the sitemaps it points at, then report which crawlers the file allows, which it blocks, and what each block keeps you out of.

The reading is done by a language model, so the result is an assessment rather than a measurement. The file we fetched is printed with it, so you can check the reading against the source.

What it checkstwo files, seven crawlers

One question, answered from the files themselves.

Blocking an AI crawler is usually an accident. It arrives in a CMS default, a security plugin, a staging config that shipped, or a line someone added in 2019 to keep scrapers out. Nothing surfaces it, because a blocked crawler leaves no trace in your analytics.

AI crawler access, named crawler by crawler

GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Googlebot, Bingbot and CCBot, each reported as allowed, blocked, partly blocked, or not named in the file at all.

Every rule, in plain words

Each User-agent, Allow, Disallow and Crawl-delay line, with what it does to the paths it covers, and a flag on the ones that block CSS, JavaScript or a content directory.

The sitemaps the file points at

Whether a Sitemap: directive exists, whether the files it names answer, how many URLs each lists, whether lastmod dates are present, and whether the XML parses.

The exact line to change

Each suggested fix comes as the directive to add or remove, not as advice to review your configuration.

How it scores

Four labels, assigned by a model, with no arithmetic behind them.

The rubric is the four-step configuration read: healthy, minor issues, significant issues, critical. Healthy means nothing in the files blocks a named crawler or breaks the sitemap. Minor issues means the setup works but is doing less than it could, usually a missing Sitemap: directive or absent lastmod dates. Significant issues means a rule is likely keeping crawlers out of content you want read. Critical means the file blocks access a site of this kind normally wants open.

There is no score out of 100 here, and there is no weighting to publish, because there is no calculation. A language model picks one of the four labels after reading the fetched files. Two runs on the same file can land on different labels, and each suggested fix carries a priority the same model chose. Treat both as a competent second opinion, not as a number to report to anyone.

What it cannot seestated before you run it

Six things this check is blind to.

An open robots.txt is the weakest possible evidence that AI can read your site. It is worth knowing, and it is worth knowing exactly how little it proves. These limits apply on every run, and they are here rather than in a footnote after the result.

Whether any crawler obeyed the file.

robots.txt is a request. Compliance is each crawler operator's own claim, and it is not observable from the file. Proving a crawler stayed out takes server logs, which this tool never sees.

Anything blocking crawlers above the file.

A CDN bot rule, a WAF challenge, an IP block or a rate limit can shut out GPTBot while robots.txt says come in. That is the more common cause in our client work, and it is invisible here.

Headers and page-level directives.

An X-Robots-Tag response header, a noindex meta tag or a canonical pointing elsewhere all override what these two files imply. This check reads robots.txt and sitemap XML, nothing else.

Whether any of it produced a citation.

Being crawlable is a precondition, not an outcome. Whether an engine names you when a buyer asks is a separate question, and it needs a prompt set run against the engines rather than a file read.

More than the first few files.

robots.txt is read to 5,000 characters, at most three sitemaps are fetched plus two children of a sitemap index, and each is truncated at 500KB. A large site's full sitemap tree is not covered.

Its own mistakes.

A language model writes every explanation and every label on the result. It can misread a wildcard or overstate a risk, so the fetched file is printed unedited underneath, and that is the version to trust.

Method

The disclosure for a tool that assesses rather than measures.

instrument: This page. An HTTP fetch of two kinds of file, read by a language model. Not Caul, and not a measurement.
what was measured: Nothing is measured. The files are fetched, then a model states what each rule does, which of seven named crawlers the file lets in, and what to change.
how: One fetch of /robots.txt at the domain root, then up to three sitemaps taken from the Sitemap: directives or, failing that, /sitemap.xml and /sitemap_index.xml, plus up to two child sitemaps of an index. URL counts and lastmod presence are counted in code; everything else is written by the model.
over what window: The moment you press the button. It is a snapshot, and a CMS update, a plugin or a security rule can change these files without anyone being told.
what this cannot tell you: Whether a crawler obeyed the file, whether a CDN or firewall blocks it anyway, whether a header or a meta tag overrides it, whether any page was indexed, and whether an engine ever named you. The model can also misread a rule, so the fetched file is printed unedited beside the reading.

Crawlable is the floor, not the finding.

This check tells you the door is open. Whether AI names you when a buyer asks for your category is a different question, and the free AI visibility report answers it with a real capture: a prompt set run across the engines, counted against competitors you name, delivered on a call. You keep the data either way.

What is in the free report All free tools