llms.txt & AI-crawler robots.txt generator
Decide which AI crawlers can read your site and point them at your best content. Toggle each bot to allow or block, add your key pages, and copy a ready-to-paste llms.txt and robots.txt. No signup.
Add the llms.txt to your site root (/llms.txt) and merge the robots.txt rules into your existing /robots.txt. On WordPress, CiteTrack AI can publish and maintain llms.txt for you, and shows which AI bots actually visit in crawler analytics.
What is llms.txt, and why add one?
llms.txt is a simple, emerging standard: a plain-text file at the root of your site that tells AI models what your site is about and points them at the content you most want them to read and cite. Think of it as a curated map for large language models, the way a sitemap is a map for search crawlers. It does not force an engine to do anything, but it gives cooperating models a clean, authoritative summary instead of leaving them to guess from your navigation and footer.
It pairs naturally with robots.txt. Where llms.txt is an invitation that highlights your best pages, robots.txt is the access control layer that decides which automated agents may fetch your site at all. Used together, you steer AI toward the content you want represented and away from the parts you do not.
Allowing vs blocking AI crawlers
Not all AI bots do the same job, and that distinction matters for this decision. Some crawlers gather data to train future models. Others fetch your pages live, at the moment a user asks a question, to ground and cite an answer. Blocking the first group protects your content from training corpora; blocking the second group removes you from the answers themselves, which is usually the opposite of what you want if you care about visibility.
That is why the generator separates them and offers a recommended preset: block the training-oriented crawlers while keeping the live answer crawlers open, so you stay citable in AI answers without feeding model training. You can override any bot individually, because the right call depends on whether your priority is reach or control.
The main AI crawlers to know
- GPTBot: OpenAI’s crawler used largely for training. OAI-SearchBot and ChatGPT-User handle live ChatGPT search and browsing.
- ClaudeBot: Anthropic’s crawler. Claude-Web / Claude-User cover live fetches.
- PerplexityBot, powers Perplexity’s answer engine and citations.
- Google-Extended, controls whether your content trains Gemini, separately from Googlebot indexing and AI Overviews.
- Applebot-Extended, Bytespider, CCBot, Meta-ExternalAgent, other training and data crawlers you may want to gate.
Whatever you choose, the only way to know it is working is to watch who actually shows up. CiteTrack AI verifies AI bot visits against published IP ranges and reverse DNS, so you can see in crawler analytics whether GPTBot, PerplexityBot and the rest are really reading your pages, and whether they respect the rules you set here.
How to use the files
Copy the llms.txt and save it as /llms.txt at your domain root. Take the robots.txt block and merge it into your existing /robots.txt, keeping any rules you already have. Re-check after a day or two: a control file is only as good as the crawlers that honour it, so confirm the behaviour you intended is the behaviour you see.
More free AI tools
Frequently asked questions
What is an llms.txt file?
llms.txt is a plain-text file at your site root that summarises what your site is about and links to the pages you most want AI models to read and cite. It works like a sitemap, but for large language models rather than search crawlers.
How do I block GPTBot or ChatGPT from my site?
Add a robots.txt rule disallowing GPTBot. This generator does it for you: toggle GPTBot to block and copy the robots.txt output. Note that blocking GPTBot (training) is different from blocking OAI-SearchBot and ChatGPT-User, which fetch pages live to cite them in answers.
Should I block AI crawlers or allow them?
It depends on your goal. Blocking training crawlers protects your content from model training. Blocking live answer crawlers removes you from AI answers, which usually hurts visibility. The recommended preset blocks training bots while keeping answer bots open.
Does llms.txt or robots.txt guarantee AI behaves?
No. Both are voluntary standards that well-behaved crawlers honour, but they are not technical enforcement. The reliable way to know what is happening is to monitor actual AI bot visits, which CiteTrack AI does with verified detection.
Where do I put the files?
Save llms.txt as /llms.txt at your domain root, and merge the robots.txt rules into your existing /robots.txt. On WordPress, CiteTrack AI can generate and maintain llms.txt automatically.
Is this generator free?
Yes, it is free with no signup. Everything runs in your browser; nothing you type is sent anywhere.
Control AI crawlers, then see who actually shows up.
CiteTrack AI maintains your llms.txt and verifies every AI bot visit inside WordPress, for a flat yearly price.
Explore: Features · Crawl Academy · Glossary · Compare · Pricing