Get Started
Start here: prove what AI crawlers actually did · Lesson 01

Crawl logs are the receipt

Use raw crawler evidence to separate what happened from what your tools assume happened.

25 minWorked exampleField assignment
0%course
What you will leave with

A one-page crawler receipt showing the exact URL, bot, status, verification state and next diagnosis.

Crawler logs are the receipt for AI visibility work. A dashboard can summarize, a prompt test can show an answer, and a screenshot can tell a story. But the log tells you the uncomfortable little truth: who requested which URL, when it happened, what your site returned, and whether the visitor was actually the bot it claimed to be.

What the log proves

Before an AI system can cite, summarize or recommend your page, something has to retrieve it. That retrieval chain is smaller than most people think: a bot arrives, proves or fails identity, requests a URL, receives a response, reads the HTML, and either uses or ignores what it found. If you skip that chain, you end up prescribing fixes from vibes.

Bot arrivesIdentity verifiedURL requestedStatus returnedHTML readAnswer uses itCitation appearsOutcome measured

Worked example: absent from ChatGPT

A SaaS company notices ChatGPT names three competitors for “best AI analytics plugin for agencies,” but does not cite them. The tempting fix is to rewrite the feature page. The crawler-first fix is to prove what happened first.

QuestionEvidenceWhat it meansNext move
Did OpenAI request it?OAI-SearchBot fetched /features/ai-analytics/ yesterdayAccess existsDo not start with robots.txt
Was it verified?Reverse DNS passedReal botTrust the request
What response?200, but answer content loaded later by scriptRender problemMove key facts into HTML
Who did AI cite?G2 and one competitor comparisonTrust/source gapImprove proof and source coverage

The receipt template

URL tested:
AI engine / crawler:
Crawler family: training / live-answer / user-triggered
Verification: verified / unverified / spoofed / unknown
Status code:
Returned content: useful HTML / thin HTML / blocked / error / redirected
Prompt or buyer question connected to this URL:
Current AI result: cited / mentioned / absent
Likely broken stage:
Next test:

What different businesses should check first

EcommerceProduct/category pages, faceted URLs, variant parameters, price and availability text in HTML.
SaaSFeature pages, pricing pages, comparison pages, docs and integration pages that answer buying objections.
Law firmPractice-area pages, location pages, attorney bios, case-result pages and pages blocked by aggressive security rules.
PublisherOriginal research, evergreen explainers, author pages, archive templates and syndication/feed crawl waste.

Do not confuse these signals

Weak conclusion

“We saw GPTBot once, so OpenAI knows our site.”

A single unverified line does not prove useful retrieval.

Useful conclusion

“Verified OAI-SearchBot fetched the pricing page, received 200 HTML, but the comparison table was missing from source.”

Now the fix has a location.

The rule

Never report “AI crawled us” without saying which crawler, whether it was verified, what URL it requested and what response your site returned.

CCiteTrack AIv1.79.66
Where this shows up in the plugin
HomeTasksVisibilitySourcesCrawlersOptimizeAnalyticsAttributionReports

Read the crawler log inside CiteTrack

The Crawlers workspace shows the request, bot family, verification state, URL and response code so the lesson starts from evidence instead of a guess.

Verified crawlers12,308official network
User-agent only138needs caution
Possible spoof34separate bucket
Access health86/10030 days

OAI-SearchBotVerified / 200/features/ai-analytics/

PerplexityBotVerified / 200/pricing/

GPTBotUser-agent only / 403/guides/ai-search/

Unknown GPTBotPossible spoof / 429/wp-login.php

Use it in this lesson

Filter to the page and crawler you care about.

Use verified rows for reporting and unverified rows for diagnosis.

Open repeated failures as tasks instead of changing policy from one log line.

Do the work

Field assignment

Scenario

Your team believes ChatGPT cannot see your pricing page because it is absent from answers. Prove or disprove that before changing the page.

  1. Find one URL and one AI engine you care about.

  2. Check whether the relevant crawler requested that URL.

  3. Record the status code, response type and verification result.

  4. Decide whether the next fix belongs to access, rendering, content or trust.

Definition of done

A one-page crawler receipt showing the exact URL, bot, status, verification state and next diagnosis.