Why AI crawler access is the first thing to check

Why access comes before every other question

Every figure in AI visibility rests on an assumption that almost nobody states out loud: that the engine could read the page. A citation rate, a share-of-voice percentage, an AI Overview presence rate all describe what an engine did with your content. None of them can tell the difference between content an engine read and passed over, and content that never reached it. Both produce the same output, which is your absence from the answer, and they call for opposite work.

That is the argument for putting access first. It is the only part of this problem where the answer is knowable with certainty, from outside the site, in minutes, with no sampling and no run-to-run variance to average out. A crawler either received a 200 with your copy in the response body or it did not. Everything downstream of that fact is probabilistic and needs repetition before it means anything at all.

The stronger version of the claim, and the one worth disagreeing with: a citation figure measured against a site the engine cannot read is not a low score, it is a broken instrument, and publishing it as a score is a category error. We already hold figures back for run-count reasons. Access failure is the harder case of the same discipline, because the number that comes out looks entirely plausible. Zero percent on a firewalled site and zero percent on a site with a genuine authority problem occupy the same cell in the same spreadsheet, and only one of the two is a finding.

It is also the cheapest finding available in an engagement, and cheap findings compete badly for attention against expensive ones. An access failure quietly produces the number that supports whatever the seller wants to sell next: a content programme, an authority programme, a retainer. Checking it first is how you find out whether the expensive diagnosis is even available to you yet.

Three layers, and only one of them is a file

Blocked has three meanings, and conflating them is the reason this gets missed. The first is robots.txt: a text file at a known path that a well-behaved crawler reads and obeys. The second is the edge: a CDN or firewall rule that answers the request with a 403 or a challenge page before it ever reaches your application, regardless of what the file says. The third is rendering: the request succeeds, the status is 200, the response arrives, and the body contains a shell with none of your content in it.

Only the first is visible in the artefact most audits inspect. Reading robots.txt tells you about robots.txt. It tells you nothing about whether your host enabled an AI bot ruleset on your behalf during onboarding, and nothing about whether your framework paints the page after the response has already been sent. The other two failures return a status code that looks correct.

Rendering failure is the worst of the three, because it is invisible from both directions at once. A person opening the page in a browser sees everything. The server log shows a 200 and a normal response size. Nothing in either view reports a problem, and the content genuinely exists for every human who ever looks at it. It simply does not exist in the bytes the fetcher received.

One check covers all three, and it is not a tool purchase. Request the page with the crawler's user-agent string and read what comes back. The status code settles layers one and two. Searching the returned body for a sentence you know is on the page settles layer three. Anything more elaborate than that is refinement, not a different method.

Which token gates which behaviour?

Each operator runs more than one crawler, and they do different jobs under different names. OpenAI's GPTBot collects for training. OAI-SearchBot builds the search index ChatGPT retrieves from. ChatGPT-User is the live fetch that happens when a person asks about a specific page in a conversation. Anthropic splits along similar lines between ClaudeBot and Claude-User. Perplexity runs PerplexityBot for its citable index and Perplexity-User for live fetches. Google-Extended is a control over Gemini app and Vertex AI training and grounding. Common Crawl's CCBot feeds a shared open corpus that an unknown set of model builders ingest.

Two consequences follow, and a robots.txt file can get both backwards without anyone noticing. The first: blocking GPTBot does not stop ChatGPT from citing you, because training and retrieval run through different pipelines under different agent names. The block that actually removes you from ChatGPT's answers is OAI-SearchBot, and a catch-all rule can hit it without a single person having decided to.

The second: Google-Extended does not govern AI Overviews, and it never has. AI Overviews are generated over Google's Search index, and that index is built by the ordinary search crawler. Google-Extended controls a different surface. Anyone who has told a client that blocking Google-Extended keeps them out of AI Overviews has sold them a change that does nothing to the surface they were worried about, and one real thing somewhere else.

The clocks differ too, and the clock is what decides whether a mistake is recoverable. A retrieval block shows up within days and reverses within days once the file changes. A training block behaves in the opposite way in both directions: it does not remove what a model has already learned about you, and it forfeits every future model generation that would otherwise have learned you. The cost compounds, the benefit is nil, and there is no undo that reaches backwards.

There are three defensible reasons to block a crawler and none of them is a feeling about AI. The content is the product, meaning a paywalled archive or a corpus you licence, in which case you disallow the training agents and keep the retrieval ones. There is a legal or contractual restriction, in which case the correct instrument is an authentication wall rather than a text file. Or you have measured the load in your own logs, in which case you rate-limit at the edge and still do not disallow. Everything outside those three is a defensive reflex that costs citations and returns nothing.

Be precise about what the file is, too. robots.txt is advisory. It works because the operators who publish crawler names also honour it. It does not stop scraping, it does not remove you from a model that has already learned you, and a disallowed URL can still be indexed by URL alone. It is a published request, not a lock.

The JavaScript trap, and why it is per-engine

There are three classes of machine that fetch your pages on behalf of AI systems, and they have different capabilities. Search-engine crawlers such as Googlebot and Bingbot render: they run a browser, execute the script, and see the finished DOM. Third-party LLM crawlers such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and CCBot fetch HTML and stop there. Browser agents, the ones that click and book and buy on a user's behalf, are browsers by construction and see whatever a person would.

For the middle class, whatever your framework paints after load never arrives. Not ranked poorly, absent. There is no second wave for them, no rendering queue, no deferred pass that picks the content up later. The raw HTML response is the entire universe of what that engine will ever know about the page.

The blanket line that gets repeated, that AI crawlers do not execute JavaScript, is wrong as stated, and the error costs money in a specific way. It is false for Googlebot, which is the engine behind AI Overviews, and false for browser agents. Generalising across the three classes turns a precise diagnosis into a vague one. The question is always per engine.

The precise version is also the useful one, because it turns a symptom into a diagnosis. A client-side rendered site can hold AI Overview citations and be entirely absent from Perplexity at the same time. Read as a content problem, that pattern produces a content programme and months of writing. It is a rendering problem, and the fix is serving the same words in the initial response.

The delta between raw and rendered hides in predictable places: tabbed and accordion panels, mega-menus, which also removes your internal link graph from the raw HTML and with it the crawler's route to your other pages, lazy-loaded body copy, load-more pagination, reviews and ratings, pricing, product specifications, and anything sitting behind an interaction. One item on that list deserves saying plainly, because it undoes work people believe they have already done: JSON-LD injected by a tag manager is not in the response an LLM crawler reads. Schema written by a script is schema that only the engines running scripts will ever see.

The fix is unglamorous and well understood. Static generation first, server rendering next, prerendering for bots as a last resort. Serve the primary content, the title, the canonical, the meta robots directives and the structured data in the initial HTML response, every time. And do not disallow your own asset paths, the framework build directory, the static folder, the fonts, because blocking those blocks rendering for the one engine that could have rescued a client-side site.

One honest gap while you are in there. A byte limit on how much of a document Google fetches has circulated for years, attributed to a Google engineer rather than to Google's own documentation, and we have not verified it. For GPTBot, ClaudeBot and PerplexityBot no equivalent figure has been published at all, so we neither assume a cap nor assume its absence. Putting the head elements and the structured data high in the document is a hedge that costs nothing in either case, which is reason enough to do it without a number to point at.

What our own robots.txt does, and what it does not do

The worked example is the file this site serves. Eleven AI agents are named individually, each with an allow rule covering the whole site: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Amazonbot and Bytespider. One path is fenced off, the API surface. The wildcard group above them says the same thing.

Now the part that a reader could otherwise use against us. For any crawler that implements the standard correctly, those eleven groups do nothing the wildcard group has not already done. Naming each agent is a disclosure, not a mechanism. It is there because a position stated in a file a stranger can fetch is checkable in a way a position stated on a page is not, and because an accidental future block gets louder: deleting a named allow line is a visible edit in a review, and adding a broad disallow next to eleven explicit allows looks like the contradiction it is.

The disallow is narrow on purpose, and it was written as a convention before there was anything behind it. There is now: the free tools on this site post to routes under that path. The order was the right way round, though it means the file briefly described a shape the site did not yet have. What it fences is an endpoint, not a page. Every essay, tool page and service page on this domain returns to a crawler exactly what it returns to a person, which is the property the rest of this argument depends on.

Two things the file cannot do, and we would rather say them than let a reader find them. It cannot make a crawler come. Welcoming eleven agents by name is a statement of intent that costs nothing and proves nothing about whether any of them has read a word of this. And it does not bind anyone who ignores it. Whether we are actually being fetched is a question for the logs, and the answer arrives as a hit count with a date on it.

The one checkbox that undoes all of it

The hardest access failure to find is the one nobody made a decision about. Managed firewall rulesets ship with categories, and one of the categories is AI bots. The platform this site deploys on, Vercel, offers a managed AI Bots ruleset that blocks GPTBot, ClaudeBot, PerplexityBot and the rest of that class. Enabling it is one toggle in a dashboard, and the result is a site that is technically immaculate, fully server-rendered, explicitly welcoming in robots.txt, and unreadable by every engine whose answers you are trying to appear in.

So the standing rule in our own operations documentation, written 2026-08-16, is that the ruleset is never enabled on this domain. Bot Protection gets the same treatment for a related reason: it challenges unverified bots, and a challenge page is not something a crawler solves. Anything in that family is scoped to the API surface and never to page routes, and any rule that could match a page route is staged in log mode first so its blast radius is observed rather than assumed. The rules we have drafted for the tool endpoints are scoped to a path robots.txt already disallows, so nothing in that work touches crawling at all.

Generalise it: the block may not be in the file. Hosts, CDNs and security plugins can enable AI bot blocking by default or on a customer's behalf, and the person responsible for the content is not necessarily told. A migration can flip it. A plugin update can flip it. None of those events announce themselves in language a marketing team reads, and the symptom is a slow disappearance from answer engines with no corresponding change to anything anybody was working on.

Name the incentive underneath this rather than resenting it. An edge vendor's job is to reduce unwanted traffic, and by that vendor's default definition an AI crawler is unwanted traffic. The default is reasonable for most of the web. It is exactly wrong for anyone whose plan is to be cited. Disagree with it deliberately and in writing, rather than inheriting it from a signup flow.

How to check, and how to know your check worked

Start with status codes, one request per agent. Fetch the same URL with GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Googlebot, Bingbot and an ordinary browser string, and print the code each one gets. Any non-200 that the browser string does not also get is an edge block, and you have located the layer without a dashboard, an account or a vendor. Do it per user-agent rather than once, because differential treatment by agent is the whole thing you are looking for.

Then diff raw against rendered. Fetch without executing any script and search the response for a sentence you know is in the body copy, then compare against the rendered DOM in a browser. Anything present in the rendered version and absent from the raw response is invisible to every LLM crawler in the middle class. For a whole site rather than a page, run a crawl with rendering off and the same crawl with rendering on, and read the difference between the two exports. That diff is the old JavaScript SEO test with new stakes attached.

Then the logs, because they are the only place AI crawler hits appear. JavaScript analytics cannot see these crawlers by construction: the tag fires when a browser runs the script, and these agents do not run scripts. A product analytics dashboard reporting no AI crawler traffic reports that on every site, including the ones being crawled hardest, and it will keep reporting it forever. Zero on an instrument that has no path to producing a non-zero is not a measurement, it is the blind spot drawn as a chart.

Verify the bot before believing the log entry. A user-agent string is a claim the requesting client makes about itself, and it is trivially spoofed. Reverse-lookup the requesting IP address, confirm the hostname sits in the operator's own domain, then forward-lookup that hostname and confirm it resolves back to the original IP. A mismatch is a spoof. An unverified crawler hit rate is not evidence, and it does not belong in a client deliverable with a label that implies it is.

What to pull, once you can read the logs at all: hit count per agent over the last thirty days, status codes per agent, which URLs each agent actually fetches, and the first date each one appeared. Those are four different findings. Zero hits from an agent across a month means you are not in its rotation. A wall of 403s means an edge rule. Being fetched regularly and still not being cited is a genuine visibility problem, and it is the only one of the three where content work is the right answer.

What access cannot tell you

Access is necessary and nowhere close to sufficient. Being fetched does not put you in the retrieval pool for any particular question, and being in the pool does not mean you are named on any particular run. Access is the floor underneath both of those, which makes it the cheapest thing to be certain about and the least likely to be the entire answer. Anyone selling a crawler audit as an AI visibility strategy is selling a floor as a building.

The sequencing argument is what carries the weight. Access goes first because it is falsifiable in minutes and because it invalidates every measurement stacked above it. If access is clean, a low citation rate is a real reading, and the diagnosis moves to the two things that are expensive: whether you have the authority to be in the pool, and whether your pages are structured well enough to be selected once you are. If access is broken, every figure produced before the fix is a description of the block wearing the label of a score.

That sets a discipline for what happens after the fix. Restart the measurement rather than continuing the line. A trend drawn across an access change is two different experiments plotted as one series, and the step in the middle will be read as progress by whoever sees the chart next. The baseline after the fix is the first honest baseline, and the work is judged from there.

What we can defend about our own configuration is narrow, and the narrow version is the one worth stating. Eleven agents named, each allowed the whole site, one endpoint fenced, the served file confirmed on 2026-08-16, and a written rule against the managed ruleset that would undo all of it in one click. No claim at all, yet, about whether any of those eleven has read this page. That answer lives in a log file, and when we have it, it will arrive as a count with a date attached rather than as a chart with a percentage on it.