Validate llms.txt, test your AI bot rules

Why test your AI crawler rules?

The rules in robots.txt are easy to get subtly wrong, and the cost of a mistake is invisible until it has already done damage. Block the wrong bot and you quietly disappear from the AI answers your buyers see. Forget to block a training crawler and your content feeds models you never intended to. Most site owners write these rules once and never check what they actually do, because there is no obvious feedback. This validator gives you that feedback: paste your files and see, in plain language, which AI crawlers you allow and which you block, and whether your llms.txt is even valid.

It matters more now because the AI crawler fleet is large and the bots are not interchangeable. A single line like Disallow: / under the wrong user-agent can mean “remove me from Perplexity’s answers” when you meant “don’t train on me.” Seeing the rules resolved per bot, the way a crawler resolves them, is the only reliable way to know your intent matches reality.

What the validator checks

llms.txt format

It confirms the basics of the emerging llms.txt standard: that the file leads with a title, includes a description or summary, and lists links in valid markdown. It flags an empty file, a missing title, or malformed links, the small mistakes that make the file useless to the models that read it.

robots.txt AI bot rules

It parses your robots.txt into its user-agent groups and resolves, for each major AI crawler, whether it is allowed or blocked at the path you choose. It applies the standard matching rules, including the most-specific-group and longest-match behaviour real crawlers use, so the verdict reflects what a bot would actually do, not a naive guess.

Allow vs block, decoded

The result separates the bots into the same groups that matter for strategy: answer crawlers that fetch pages to cite you live, training crawlers that gather data for models, and search crawlers that feed the index behind AI Overviews. Seeing your rules sorted this way makes the strategic question obvious. Most sites want answer crawlers allowed and decide deliberately about training crawlers, and the validator shows whether your file actually achieves that.

From rules to reality

One caveat worth stating plainly: robots.txt and llms.txt are voluntary. A well-behaved crawler honours them; a badly-behaved one ignores them. So testing your rules tells you what compliant bots should do, not what every bot does. The only way to know what is really happening is to watch the crawlers themselves. CiteTrack AI does that inside WordPress, logging real AI bot visits, verifying them against published IP ranges, and showing whether the rules you set here are being respected. Pair this validator with the log analyzer to check both your intent and your reality.

Frequently asked questions