The Crawl Health Score is the fast answer to one question: can useful AI crawlers reach the pages that matter? It combines frequency, coverage, error rate and response time into one 0-100 score, then shows the evidence underneath.
The main screen
The normal Crawl Health screen is designed to load quickly even on large sites. It shows the access and identity summary, recent verified activity, the most active crawler companies, important pages, response evidence and what needs attention.
Identity states
- Verified network: the request matched a known crawler and official network evidence.
- User-agent only: the name claimed to be a crawler, but network verification was not available yet.
- Possible spoof: the name looked like an AI crawler but failed verification or matched suspicious behavior.
Filters
Use the page-wide filters for date range, crawler company, identity state and URL/user-agent search. This lets you answer questions like “did ClaudeBot reach my pricing page this month?” or “are possible spoof requests hitting login pages?” without exporting the crawl log.
Advanced diagnostics
The expensive investigation charts live behind Load advanced diagnostics. Open it when you need detailed change analysis, daily crawler volume, response-time trends, error charts, page-value comparisons or the full site-wide crawl-waste analysis. Keeping that behind a click makes the everyday Crawl Health screen faster.
How to improve the score
- Fix 4xx/5xx responses on important URLs.
- Keep answer pages linked and present in your sitemap.
- Allow the AI crawlers you want in robots.txt and block only the ones you intentionally do not want.
- Improve slow response times on pages AI crawlers request often.
- Use Optimize for pages that are crawled but not cited.