Get Started
Start here: prove what AI crawlers actually did · Lesson 02

Diagnose before you block or optimize

A practical decision tree for deciding whether the problem is access, crawl waste, rendering, source trust or measurement.

28 minWorked exampleField assignment
0%course
What you will leave with

A crawl policy decision tree and a measurement plan that avoids accidental self-sabotage.

The wrong first move is picking a side: “block AI” or “open everything.” The useful move is diagnosis. Some crawlers train future models, some fetch live answers, some are search engines, and some are impostors. Treating them as one group is how teams accidentally block the traffic they wanted and allow the traffic they feared.

The crawl decision tree

Ask these in order

What role is this crawler playing?Training, live-answer search, user-triggered fetch, classic search or unknown.
Can we verify it?Verified crawlers can be measured and governed. Unverified names belong in a separate bucket.
Do we want this role?You may allow live-answer retrieval while restricting training. The policy should match the business goal.
Is the crawler reaching valuable pages?High request counts on feeds and parameters may be waste, not opportunity.
Can we connect it to citations or revenue?Retrieval is a leading signal. It becomes business evidence only when tied to answers, visits, leads or orders.

Example policy: SaaS company that wants citations, not blind training

Crawler typeExamplesPolicyReason
Classic searchGooglebot, BingbotAllowOrganic search still matters
Live-answerOAI-SearchBot, PerplexityBotAllow + monitorCan create citations now
User-triggeredChatGPT-User, Claude-UserAllow + monitorSignals real people asking about you
TrainingGPTBot, ClaudeBotDecide intentionallyBrand/policy decision, not an SEO reflex
ImpostorsAny failed verificationRestrictProtect data quality and resources

Three common diagnosis paths

High crawls, low citationsCheck crawl waste, thin pages, missing answer blocks, stale source ecosystem and whether AI cites third-party sources instead.
Low crawls, good contentCheck robots.txt, CDN/WAF rules, internal links, sitemap discovery and whether pages return useful server HTML.
Lots of “GPTBot,” no business changeVerify identity. You may be counting spoofed or training traffic as if it were live-answer demand.
Citations, no conversionsInspect prompt intent, citation position, landing page fit and attribution setup before rewriting crawler policy.

The prompt to force a better diagnosis

Act as a crawler evidence auditor.
Do not recommend a robots.txt or content fix yet.
Classify the problem as one or more of: identity, access, crawl waste, rendering, answerability, source trust, attribution.
Use only the evidence below. Label anything else as an assumption.

Business goal:
URL:
Crawler log evidence:
Robots / llms.txt rules:
Current AI answer:
Known citations or sources:
Conversion evidence:

Definition of done

  • You know which crawler roles you allow, monitor or restrict.
  • Verified and unverified traffic are reported separately.
  • Every important 403, 429, 5xx or empty-HTML issue has an owner.
  • Crawl waste is measured separately from productive crawling.
  • One tracked prompt or business outcome is attached to every major crawler fix.
CCiteTrack AIv1.79.66
Where this shows up in the plugin
HomeTasksVisibilitySourcesCrawlersOptimizeAnalyticsAttributionReports

Send repeated crawler problems to Tasks

When the same access, crawl waste or page issue repeats, CiteTrack turns it into owned work with saved evidence and activity history.

Crawler tasks9open
Access failures3verified bots
Waste patterns4URL groups
Due reviews2this week

403 on /guides/ai-search/High priorityOwner: Dev

Filters crawled repeatedlyMedium priorityOwner: SEO

Possible spoof spikeMonitorOwner: Security

Pricing page now 200ResolvedReview next scan

Use it in this lesson

Decide whether the issue belongs to Crawlers, Optimize or Sources.

Assign the repeated issue with owner and expected outcome.

Use Check again now or the next scan to confirm whether it stayed fixed.

Do the work

Field assignment

Scenario

A store owner wants to block every AI crawler because “AI is stealing content,” while the marketing team wants citations from Perplexity.

  1. Separate training, search, live-answer and user-triggered crawlers.

  2. Choose which crawler roles should be allowed, monitored or blocked.

  3. Run one diagnosis for a page that should be cited but is not.

  4. Write the policy and the measurement plan together.

Definition of done

A crawl policy decision tree and a measurement plan that avoids accidental self-sabotage.