Which AI bots are crawling your site?
Paste a chunk of your server access log and see every AI crawler that hit your site, how often, which pages they fetched, and where they burned crawl budget. It runs entirely in your browser, so your logs never leave your computer.
Everything is parsed locally in your browser. Nothing is uploaded or stored. Big files are sampled to the first ~50,000 lines.
This reads a snapshot you paste in. To watch AI crawlers continuously, verify them against published IP ranges, and turn it into a crawl-health score, see CiteTrack AI crawler analytics.
Why your server log is the ground truth
Every time an AI crawler fetches a page, your web server writes a line about it: the IP, the time, the URL, the response code and the user-agent string that names the bot. That log is the only place you can see what AI engines actually do on your site, as opposed to what they say they do. Analytics tools like Google Analytics run on JavaScript that bots usually don’t execute, so AI crawlers are invisible there. The access log is where they show up.
This tool reads that log and pulls out the AI traffic for you. Paste a few thousand lines, or upload the file, and it counts how many times each AI bot visited, which URLs they hit most, and where they spent requests on pages that will never be cited. It is the fastest way to answer the first question of AI visibility: can the engines even reach my content, and what are they reading?
How the analyzer works
It parses the common Apache and Nginx “combined” log format, where each line ends with a quoted user-agent. It matches that user-agent against a built-in list of known AI crawler signatures, groups the matches by bot, and summarises the hits, the top pages, the response codes and the obvious crawl waste. All of it happens in your browser with JavaScript, so the log never travels anywhere. Close the tab and it is gone.
The AI crawlers it recognises
The tool knows the current crawler fleet and what each one is for, which matters because they are not interchangeable:
- Answer crawlers: OAI-SearchBot and ChatGPT-User (ChatGPT), PerplexityBot (Perplexity), Claude-User and Claude-Web (Claude). These fetch pages to build and cite live answers. You want these reaching your best content.
- Training crawlers: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent. These gather data for model training. Whether you allow them is a deliberate choice.
- Search crawlers: Googlebot and Bingbot, which still feed AI Overviews and Copilot through the index.
What “crawl waste” means, and why it matters
Crawl waste is when a bot spends requests on pages that can never earn a citation: 404s and other errors, faceted or parameter URLs, internal search results, cart and checkout pages, or the same low-value URL fetched over and over. Every wasted request is attention an engine did not spend on the content you actually want represented. The analyzer flags these patterns so you can see whether AI crawlers are reading your money pages or wandering through dead ends. Fixing that, with redirects, robots rules and a clean llms.txt, is some of the highest-leverage AI-readiness work there is.
From a snapshot to continuous monitoring
A pasted log is a point-in-time check. The limitation is obvious: it is only the lines you happened to grab, it cannot confirm a bot is genuine, and it does not trend over time. A user-agent can be spoofed, so a line claiming to be GPTBot is not proof it really was. CiteTrack AI closes those gaps inside WordPress: it logs AI crawler hits continuously, verifies them against each provider’s published IP ranges and reverse DNS, rolls the result into a crawl-health score, and alerts you when something changes. See crawler analytics and the llms.txt generator to act on what you find here.
More free AI tools
Frequently asked questions
How do I see which AI bots are crawling my site?
Open your server access log (Apache or Nginx), copy a chunk of recent lines, and paste them into this analyzer. It finds every known AI crawler by its user-agent and shows hits, top pages and crawl waste. Nothing leaves your browser.
Where do I find my access log?
On most hosts it is access.log or the “Raw Access Logs” download in cPanel; on Nginx it is usually /var/log/nginx/access.log; many managed WordPress hosts expose it in their dashboard. Paste the lines or upload the file here.
Is my log data safe?
Yes. The analyzer runs entirely in your browser with JavaScript. Your log is never uploaded, sent to a server or stored. Close the tab and it is gone.
Can AI crawler user-agents be faked?
Yes, a user-agent string can be spoofed, so a single line is not proof. To confirm a bot is genuine you need to verify its IP against the provider’s published ranges, which CiteTrack AI does automatically and this snapshot tool does not.
What is crawl waste?
It is AI crawlers spending requests on pages that cannot be cited: errors, parameter URLs, search results, cart pages, or the same URL fetched repeatedly. The analyzer flags it so you can redirect or block those paths and steer crawlers to real content.
How is this different from CiteTrack AI?
This is a one-off snapshot of logs you paste in. CiteTrack AI logs AI crawler visits continuously inside WordPress, verifies them against published IP ranges, scores crawl health and alerts you to changes over time.
See AI crawlers once here. Watch them continuously with CiteTrack AI.
Verified AI bot detection, crawl-health scoring and alerts, inside WordPress, for a flat yearly price.
Explore: Features · Crawl Academy · Glossary · Compare · Pricing