Why your llms.txt and robots.txt need to agree
Most sites that have started thinking about AI now run two separate files that both speak to machines about access. The familiar one is robots.txt, the decades-old standard that tells crawlers which paths they may and may not fetch. The newer one is llms.txt, a file proposed to give large language models a curated, plain-language map of your most important content. The problem is that these two files are written at different times, by different people, for different audiences — and nobody checks that they tell the same story. This tool fetches both, parses them, and flags every place where one file invites an AI system in while the other shuts it out.
That contradiction matters because it produces outcomes you did not intend. If your llms.txt proudly lists a documentation section as a key resource for AI models, but your robots.txt blocks the AI crawler that would need to fetch it, you have published an invitation to a locked door. The model is told the content exists and is told it cannot read it. The reverse is just as damaging: a robots.txt that happily allows every AI bot while your llms.txt omits or forbids the same areas sends a mixed signal that undermines the careful curation llms.txt is supposed to provide.
What the checker actually fetches and compares
The tool requests two files from the site root: /llms.txt and /robots.txt. It reads the robots.txt the way a crawler does — grouping rules by user-agent, collecting Allow and Disallow paths, and paying special attention to the AI-specific agents like GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, and the others that have appeared as the AI ecosystem grew. It then reads llms.txt as a content manifest: the links it promotes, the sections it names as primary, and any access intent it expresses. The comparison is not line-for-line text matching; it is intent matching. The question is always the same: does each file imply the same answer to "can an AI system use this part of the site?"
Because the two files use completely different formats — robots.txt is a path-based rule list, llms.txt is a Markdown-style document of curated links — the tool has to translate both into a common notion of allowed versus blocked before it can spot a conflict. A URL featured in llms.txt is resolved back to a path and checked against the robots.txt rules that would apply to AI agents. When the verdicts disagree, that pairing surfaces as a contradiction in the results.
The contradictions it is built to catch
The headline case is the promoted-but-blocked URL: a page your llms.txt highlights as important for models, sitting under a path your robots.txt disallows for the very agents those models use. This is the most expensive mistake because it actively wastes the curation effort you put into llms.txt. The mirror case is the allowed-but-omitted area: content your robots.txt opens to AI bots that your llms.txt never mentions, which is less a hard error and more a missed opportunity to guide the model toward your best material.
It also catches agent-level inconsistency. A common pattern is a robots.txt that blocks one AI crawler — say GPTBot — while allowing another, alongside an llms.txt that treats all AI systems the same. That is sometimes deliberate, but often it is a leftover from a copy-pasted rule nobody revisited. The tool surfaces which agents are treated differently so you can confirm the split is intentional rather than accidental. It similarly flags the case where robots.txt blocks everything with a blanket disallow while llms.txt still advertises content, which guarantees the manifest can never be honored.
How to read the consistency report
The report pairs findings rather than listing two files side by side. Each contradiction names the resource or area in question, states what llms.txt implies for it, states what robots.txt implies for the relevant AI agents, and explains why the two disagree. A finding marked as a hard conflict means a path you actively promote is actively blocked — fix this first. A softer finding might note that an allowed section is absent from your manifest, which is advisory rather than urgent.
Read the agent breakdown carefully. Because robots.txt rules are per user-agent, the same URL can be allowed for one AI bot and blocked for another, and a single llms.txt cannot express that nuance. The tool tells you which agents see which verdict so you can decide whether a difference is a policy you chose or a contradiction you forgot. The goal of the whole report is a clean bill of health: every URL your llms.txt promotes should be fetchable by the AI agents you intend to serve, and your robots.txt should not silently contradict that intent.
Common mistakes that create these conflicts
The classic source of trouble is timing. Teams add an aggressive AI-blocking rule to robots.txt during a period of worry about training data, then later publish an llms.txt to court AI visibility — and never reconcile the two. The block from the old decision quietly defeats the new strategy. A related mistake is blocking AI crawlers wholesale to "protect content" while still hoping to be cited in AI answers; you cannot have both, and llms.txt makes that tension visible.
Path drift is another culprit. A site reorganizes its URLs, the robots.txt disallow rules get updated, but the links inside llms.txt still point at the old structure or at a parent folder that is now blocked. The manifest looks fine in isolation and the robots.txt looks fine in isolation; only by checking them together do you notice that the promoted links no longer resolve to crawlable paths. Finally, people forget that robots.txt only governs crawling, not every form of access, and they assume a Disallow in robots.txt is a complete AI policy. The checker keeps the two files honest with each other but it is not a substitute for understanding what each one can and cannot enforce.
Where this fits in AI-search strategy for 2026
As answer engines and retrieval systems mature, the files at your site root have become a small but real control surface for how AI sees you. robots.txt decides who may fetch; llms.txt decides what you want them to focus on once they are in. They only work as a pair. A coherent setup — robots.txt allowing the AI agents you want, llms.txt guiding them to your strongest, most quotable content — turns these two files into a deliberate front door rather than two doors that argue with each other.
Consistency between them also signals competence to the crawlers themselves. A site whose access files contradict each other looks unmaintained, and unmaintained signals rarely earn trust. Keeping the two aligned is part of the same discipline as keeping your canonical tags, sitemap, and meta robots in agreement: every machine-facing declaration about a URL should point the same direction. When they do, AI systems spend their effort reading your content instead of resolving your contradictions.
Understanding what each file can and cannot control
A lot of confusion between these two files comes from misreading what each one is for. robots.txt is an access-control file in the crawling sense: it tells well-behaved bots which paths they are permitted to request. It is a request-time gate, enforced by the politeness of the crawler, and it says nothing about what should happen to content a bot has already fetched. llms.txt is the opposite kind of file — it is not a gate at all but a recommendation, a curated table of contents that says "if you are an AI system trying to understand this site, these are the pages that matter and here is how they fit together." One restricts, the other guides.
Because their jobs differ, they fail in different ways, and the checker keeps both failure modes in view. A robots.txt mistake tends to be silent and severe: a single stray Disallow line can wall off content you desperately want AI systems to read, and nothing on the page warns you. An llms.txt mistake tends to be a mismatch with reality: it promotes pages that moved, recommends sections that are thin, or describes a structure the site no longer has. The consistency check is valuable precisely because it sits at the seam between these two failure modes, where a restriction in one file quietly cancels a recommendation in the other.
It is also worth remembering that neither file is a security boundary. A path blocked in robots.txt is still publicly reachable by anyone who types the URL, and a page omitted from llms.txt is still crawlable if robots.txt allows it. Treating these files as a tidy public statement of intent — rather than as a lock — is the right mental model. The checker's job is to make sure that public statement of intent is coherent, so that the story your site tells AI systems about access is the same story no matter which file they read first.
What to do after you run the checker
Start with the hard conflicts. For every URL your llms.txt promotes that robots.txt blocks for AI agents, decide which file is wrong. If you genuinely want AI systems to use that content, loosen the robots.txt rule for the relevant agents. If you do not, remove the link from llms.txt so your manifest stops advertising something you intend to withhold. Make the decision once, in the right file, and the contradiction disappears.
Next, settle your per-agent policy deliberately. Decide which AI crawlers you allow and apply the same logic in both files, so a bot you welcome in robots.txt is also served by the content your llms.txt points to. Then re-run the checker to confirm a clean result, and re-run it whenever you reorganize URLs, edit robots.txt, or update your llms.txt — those are exactly the moments when the two files drift back out of sync. Treat agreement between them as a standing invariant, not a one-time cleanup.