Get Started
Make the site retrievable · Lesson 04

Diagnose AI crawler access

Read server logs, distinguish training from answer bots, and fix the actual layer returning 403, 429 or 5xx errors.

28 minWorked exampleField assignment
0%course
What you will leave with

A before/after log pair showing the same bot and URL moving from failure to 200.

A robots.txt audit is not “open the file and look for Disallow.” You need to prove what the bot requested, what responded, and whether the visitor was genuine.

Read one log line completely

198.51.100.24 - [21/Jul/2026:10:00:00]
"GET /guides/ai-analytics/ HTTP/1.1" 403 150
"Mozilla/5.0 (compatible; GPTBot/1.2)"
198.51.100.24Verify the network identity/guides/ai-analytics/Confirm this page should be public403The request reached a blocking layerGPTBot/1.2A claim, not proof

The 15-minute diagnosis

  1. Confirm scope

    Is one URL failing, one directory, one bot, or every AI bot? A single URL suggests page/security rules; every URL suggests WAF or robots policy.

  2. Verify the bot

    Use forward-confirmed reverse DNS. Do not whitelist a crawler based only on its user-agent.

  3. Check each layer

    robots.txt → CDN/WAF → host firewall → WordPress security plugin → application response.

  4. Retest the exact pair

    Same bot identity, same URL, then confirm the new status in raw logs.

Response-code decision tree

403Permission or security rule. Check WAF, bot protection, plugins and robots policy.
429Rate limiting. Lower crawl pressure or create a verified-bot exception.
5xxOrigin failure. Check PHP errors, timeouts, memory and upstream availability.
200 but emptyRendering/extraction problem. Inspect returned HTML, not the browser screenshot.

Three situations, three fixes

Cloudflare blocks verified botsCreate a narrow allow rule using verified-bot signals, not a global bypass.
Security plugin challenges all non-browser trafficExclude verified crawler networks and keep rate limits for unknown agents.
Store filters consume the crawlBlock parameter combinations and point bots at canonical category/product URLs.
Do not promise llms.txt will fix access.

It is a map, not a permission layer. A blocked or failing URL stays blocked or failing.

Do the work

Field assignment

Scenario

GPTBot receives 403 on your articles, but robots.txt says it is allowed. The block is somewhere else.

  1. Find the response layer: WordPress, security plugin, CDN/WAF or origin server.

  2. Test one affected URL with the bot user agent and record the response.

  3. Change one rule, retest, and verify a clean 200 response in the logs.

Definition of done

A before/after log pair showing the same bot and URL moving from failure to 200.