Reduce crawl waste with robots.txt

Crawl waste is bot activity on URLs that should not be crawled, error pages, feeds, internal search, tracking-parameter URLs. The Crawlers screen buckets recent waste and generates a robots.txt block.

1. Open CiteTrack AI → Crawlers and review the Crawl Waste table.
2. Copy the suggested robots.txt additions (it skips rules you already have).
3. Add them to your robots.txt, or set the AI Visibility Mode under Settings.

← All articles