What is Crawl budget?
The finite effort a bot spends on your site.
Crawl budget is the limited amount of fetching a crawler will do on your site in a given period. Spend it on error pages, feeds, search results and parameter duplicates, crawl waste, and the pages you actually want cited may go unread.
CiteTrack buckets recent waste and generates a targeted robots.txt block so AI crawlers concentrate on content that can earn citations.
Why — and when — does it matter?
Crawl budget matters when valuable URLs are revisited too slowly while bots spend requests on errors, duplicates, parameters or low-value archives.
How it works
Verify the crawler and count requests.
Classify valuable pages and waste patterns.
Inspect status codes, parameters and crawl depth.
Improve links, canonicals and targeted access rules.
Monitor whether useful-page coverage and recency improve.
Worked example
A store receives 20,000 AI-bot hits, but 70% target filtered URLs. Canonicals, internal links and careful blocking redirect attention toward category guides and products.
How to track and report it
Reporting rule: State the engine, prompt set, market, time window and denominator. A number without scope cannot be reproduced or compared.
Practical playbook
- Verify the crawler and count requests.
- Classify valuable pages and waste patterns.
- Inspect status codes, parameters and crawl depth.
- Improve links, canonicals and targeted access rules.
- Monitor whether useful-page coverage and recency improve.
Common mistakes
- Blocking parameters without testing
- Using raw hits as success
- Applying large-site crawl assumptions to every site
Questions people ask
What is Crawl budget?
The finite effort a bot spends on your site. Crawl budget matters when valuable URLs are revisited too slowly while bots spend requests on errors, duplicates, parameters or low-value archives.
How does Crawl budget work?
Verify the crawler and count requests. Classify valuable pages and waste patterns. Inspect status codes, parameters and crawl depth. Improve links, canonicals and targeted access rules. Monitor whether useful-page coverage and recency improve.
How do you measure Crawl budget?
Useful-page crawl coverage and waste rate. Days since last crawl and error-request rate. Waste rate = low-value or error requests ÷ verified requests × 100
What is a common mistake with Crawl budget?
Blocking parameters without testing. The scope and methodology should be stated before interpreting the result.