Get Started
Crawl Academy

Diagnose how AI finds, reads and cites your site

A guided field course for identifying real AI bots, fixing crawler access, eliminating crawl waste and connecting retrieval to citations. Every lesson ends with a concrete assignment and definition of done.

3 modules12 field lessonsSaved progressWorked examples
Start the course
0%complete
Learn → apply → measure → improve
Start here

Crawler logs are the receipt. They show what actually happened.

Most AI visibility advice starts at the content layer. Crawl Academy starts one step earlier: did the right AI system reach the right page, receive usable HTML, and come back often enough for your work to matter?

The course teaches the operating loop in order: separate real bots from claims, prove access, remove crawl waste, keep permission files aligned, then connect retrieval to citations, referrals and revenue.

1. IdentifyWhich AI or search crawler arrived?
2. VerifyWas it genuine, unverified or spoofed?
3. DiagnoseDid it receive a useful response?
4. ImproveRoute crawl effort toward citable pages.

Your course path

Complete the lessons in order. Each one ends with a field assignment and a definition of done.

0 / 12 complete
Module 01

Start here: prove what AI crawlers actually did

Before changing robots.txt or rewriting pages, use the crawler record to see which bots arrived, whether they were verified, what they requested and what your site returned.

✓01Crawl logs are the receiptUse raw crawler evidence to separate what happened from what your tools assume happened.25 min✓02Diagnose before you block or optimizeA practical decision tree for deciding whether the problem is access, crawl waste, rendering, source trust or measurement.28 min
Module 02

AI crawlers, end to end

Who reads your site for AI, how to tell the real bots from impostors, and how to stop them wasting effort.

✓03What is an AI crawler?The bots that read your site to train and answer for AI engines, and why they decide whether you ever get cited.15 min✓04GPTBot, ClaudeBot & PerplexityBot explainedA field guide to the AI bots you’ll see most in your logs, and what each one means for you.15 min✓05How to verify a bot is real (not a spoof)Any script can claim to be Googlebot. Here’s the only reliable way to confirm it.15 min✓06What is crawl waste?When AI bots spend their effort on the wrong URLs, and what it costs you.15 min✓07robots.txt, llms.txt & ai.txt, controlling AI accessThe three text files that tell AI what it may crawl, train on, and feature, and which actually matter.15 min
Module 03

Getting cited & measured

How engines pick sources, how to be picked, and how to measure whether it’s working.

✓08How AI decides what to citeThe retrieval-then-synthesis pipeline behind every cited link, and where you can influence it.15 min✓09AEO vs GEO vs SEOThree optimizations, three goals, and why you now need all three.15 min✓10Cited vs mentioned vs absentAll three are different results. Only one sends a clickable link.15 min✓11Share of voice, explainedA single number for how much of the AI conversation you own.15 min✓12What is llms.txt? (and does it work)An emerging convention worth shipping, with realistic expectations.15 min

Turn crawler evidence into action

CiteTrack AI verifies the bots, records every request and connects crawler behavior to citation and revenue outcomes.

Get CiteTrack AI

Explore: Features · Crawl Academy · Glossary · Compare · Pricing