How often AI crawlers revisit a page after you update it

How fast AI crawlers come back after an update

Update a page and AI crawlers usually come back within hours to a few days, not weeks. In Conductor's case study, within 24 hours of a July 24 publish, Perplexity matched Google's mobile crawl count and ChatGPT hit the page three times. Low-authority pages wait longer. High-traffic pages get revisited fast.

The gap between engines is wide. In one log sample taken five days after publish, ChatGPT had crawled a page roughly eight times more often than Google, and Perplexity about three times more often. In some cases AI crawlers visited pages over 100 times more than Google or Bing combined.

The reason is volume and appetite. Daily AI-agent requests to Cloudflare's network jumped over 1,700% from June 2025 to May 2026. A large share of all crawler requests now comes from AI training bots rather than search engines. More crawlers hitting your site more often means a shorter wait after you change something.

Here's the part that trips people up. A crawler coming back is not the same as your change showing up in an answer. A bot can fetch your new HTML today while ChatGPT, Gemini, or Perplexity keeps quoting last month's version from a cached index. The revisit is step one. Whether the engine reads, re-indexes, or live-fetches the fresh page decides when the answer actually updates.

AI crawlers vs Googlebot: why they come back so much more often

An AI crawler fetches your page to feed a large language model, either to train it or to answer a live question. A traditional search crawler like Googlebot fetches your page to build one giant ranked index. Same act, different purpose. That difference in purpose is why the visit frequencies look nothing alike.

Google's index held roughly 400 billion documents as of 2020, the last figure Google has disclosed, revealed under oath in the US antitrust trial. To keep that index current, Googlebot budgets its visits carefully. A low-traffic page might get re-crawled every few weeks. AI crawlers work under no such restraint.

The trend is steep across the whole network. Cloudflare also tracked a 35 to 40% drop in human traffic across retail, computer software, IT services, and financial services from June 2025 to April 2026. Fewer people clicking through, more bots reading on their behalf.

High crawl frequency does not mean fast answer updates. A crawler can fetch your revised page ten times a day and the model can still quote your old numbers, because the visit and the answer are two separate systems. What the bot fetched has to survive a caching layer, a re-index step, and in some cases a training cutoff before it shows up in a reply.

The reassuring takeaway: crawling is rarely your bottleneck. AI bots come back constantly, often before Google notices you changed anything. If your update isn't landing in answers, the delay is almost always downstream of the crawl.

Engine by engine: how ChatGPT, Gemini, Perplexity, Claude, and Copilot handle a re-crawl

The engines pick up an edit at different speeds because they read your page in different ways.

Perplexity usually reflects an update fastest because it fetches pages live at query time. In the same 24-hour test window, Perplexity had crawled a new page as many times as Google's mobile crawler. If your update is a number, a date, or a short claim Perplexity can quote directly, it tends to surface there first.

ChatGPT runs two separate systems. The training crawl (GPTBot) revisits on its own cadence. But when a user asks about your topic and ChatGPT browses live, it fetches the current page in that moment. So an edit can show up almost immediately in a live-browsing answer while the cached version behind non-browsing responses still shows your old copy. Same brand, same day, two different answers.

Gemini leans on Google's index and structured data. It rarely fetches your page fresh on its own. If Googlebot hasn't re-crawled since your edit, Gemini is still working from the stale index entry. Your Gemini freshness is basically your Google freshness.

Claude relies primarily on training data with limited web access. Updates take longer to reflect because they depend on Anthropic's training schedule rather than live retrieval.

Copilot uses Bing's search index. Update speed tracks Bing's re-crawl schedule, which is typically faster than Google for recently modified pages. Bing Webmaster Tools shows grounding queries and direct citations.

Check Perplexity first when you want to confirm an edit actually landed. If Perplexity has it and Gemini doesn't, that's not a bug. That's Google's crawl queue lagging.

Crawl, live fetch, or cached index: how AI systems detect a change

An AI system notices your update through one of three retrieval modes, and which one it uses decides how fast the change shows up.

Scheduled crawl (slow lane): a bot fetches your page on its own timing and stores what it sees. It compares what it fetches now against what it stored last time. If the HTML differs, the new version replaces the old copy in the index. The bot decides when to come back, not you.

Live fetch (fast lane): the engine grabs your page at the moment someone asks a question. Your update is visible almost immediately because there is no stored copy sitting between your edit and the answer. The catch is that not every query triggers a live fetch.

Cached index (the lag): the answer comes from a stored copy that may be hours or weeks old. Until the next scheduled crawl overwrites it, the AI keeps quoting the old version even though your live page says something different.

Two signals help a scheduled crawler notice your edit sooner. The first is your sitemap's lastmod date. When it moves forward, a bot has a reason to re-fetch instead of trusting its cached copy. The second is broader freshness signals: new internal links pointing at the updated page, a fresh publish date in the HTML, or the page appearing in an updated feed.

Why the AI still shows your old content after you updated the page

The AI shows old content because a re-crawl and a fresh answer are two different events, and one of three layers is probably serving the previous version.

Layer 1: Your CDN or page cache. WordPress caching plugins (WP Rocket, W3 Total Cache, LiteSpeed) and CDNs like Cloudflare store a static copy of your page's HTML and hand that to whoever asks, including AI crawlers. When you hit update, the browser you're logged into may see the change instantly while the cached copy sits unchanged for hours. Purge the cache for that URL and you rule this layer out.

Layer 2: The AI system's own retrieval index. Even after a crawler fetches your fresh page, that data has to replace the old version in the model's index. This is the layer you can't purge. You wait, or you push freshness signals.

Layer 3: JavaScript-loaded content. Most AI crawlers read raw HTML only. They do not run JavaScript the way a browser does. If your update lives in a block that loads client-side (a dynamic pricing table, a "read more" toggle, content injected by a page builder), the crawler sees an empty container where your new text should be.

Check them in this order, cheapest first:

1. View source, not the rendered page. If your new text isn't in the raw HTML, it's the JavaScript layer. Fix that first.
2. Fetch as an outsider. Open in a private window. If it shows the old version, it's the CDN cache. Purge it.
3. Confirm the crawler returned. Check server logs for AI user-agents with a timestamp after your edit.
4. If HTML is correct, cache is purged, and the crawler came back, it's the AI system's own index. You're waiting on their rebuild.

How to check whether AI crawlers actually saw your new version

A crawl hit in your logs is the only proof that an AI bot fetched your new version. Here is the check, step by step.

1. Open your server or WordPress access logs. On managed WordPress hosts this is usually under "Raw access logs" or "Analytics."
2. Filter for AI user-agents. Search for GPTBot and ChatGPT-User (OpenAI), PerplexityBot, Google-Extended (Gemini), ClaudeBot, and Amazonbot. Note that GPTBot handles scheduled training crawls while ChatGPT-User fires during live retrieval.
3. Compare the hit timestamp to your edit timestamp. A bot line dated after you saved the change confirms the crawler requested the page again. Match the exact path, not just the domain.
4. Check the served HTML with view source. If your new text only appears after JavaScript runs, most AI crawlers won't see it.
5. Run the same prompt across engines. Perplexity tends to reflect a fresh fetch first. If one engine shows the new fact and another shows the old one, the difference is retrieval mode, not your page.
6. Log which engine flips and when. That gap between your edit and the first correct citation is your real revisit window.

Doing this by hand for one page is manageable. Doing it every time you publish, across five engines and hundreds of URLs, is where it falls apart. CiteTrack's crawler log analyzer watches your logs continuously and flags which AI bots hit which page after each edit. The Chrome extension scores any page for crawlability and structural readiness. And citation tracking runs your prompts across five engines on a schedule so you see when an answer actually updates.

How to force a faster re-crawl after you update a page

You control more of the re-crawl timeline than most people assume. Work through these steps in order.

1. Update the sitemap lastmod to the real edit date. Confirm the lastmod value for the edited page matches today, not the original publish date. Most WordPress SEO plugins do this automatically, but caching can freeze the sitemap.
2. Clear your CDN and page cache after editing. Purge the cache for that specific URL, then load the page in an incognito window to confirm you see the new text. This single step resolves most "I updated it but the AI still shows the old version" cases.
3. Add internal links from your most-crawled pages. Link to the edited page from your homepage or a popular blog post. Crawlers discover changes by following links from pages they already visit often.
4. Keep the changed content in raw HTML, not JavaScript. Confirm the revised text appears in the page source. If it only shows up after scripts run, many AI crawlers will miss it.
5. Add genuine freshness signals to the body. Show a visible "Updated on" date near the top, and actually revise the copy. A changed timestamp with identical text is a weak signal.
6. Update dateModified schema and llms.txt. Both give AI systems a clean, machine-readable pointer to what changed. CiteTrack's free Schema Markup Generator and llms.txt Generator check that these are formatted correctly.

Skip the meta revisit-after tag. There is also no "recrawl now" button for ChatGPT or Perplexity, so these six levers are the real toolkit.

The revisit-after meta tag does nothing (and other dead ends)

The meta name="revisit-after" tag does nothing. Google has said since 2007 that it is not used by Google or any other major search engine, and no AI crawler honors it either. If you added it hoping to tell crawlers "come back in 3 days," you told nobody anything.

The second dead end is the manual re-crawl button people expect for AI bots. Google Search Console has URL inspection, where you can request indexing. There is no equivalent for AI crawlers. You cannot paste a URL and ping ChatGPT or Perplexity to re-fetch it.

Other placebo controls to skip:

• Priority in your XML sitemap. The priority value is a hint most crawlers now discount entirely.
• Changefreq in the sitemap. Setting changefreq=daily does not make anything crawl daily. Crawlers learn your real update pattern and ignore your stated one.
• Meta refresh and cache-busting query strings. These affect what a browser shows a human, not what a crawler schedules.
• Resubmitting the same sitemap over and over. One accurate lastmod date does far more than ten resubmissions.

Every minute you spend on revisit-after or sitemap priority is a minute your stale answer stays live for anyone asking ChatGPT about you.

When chasing a faster crawl isn't worth it

Not every page deserves the effort. A decision rule worth writing down:

• High-authority page, real traffic, factual update (pricing, product specs, corrected stat): worth speeding up.
• Low-authority page, little traffic, cosmetic edit: not worth it. Update it, submit your sitemap, move on.
• Answer looks stale but logs show a fresh fetch: don't touch crawl settings. Fix the cache instead.

Sometimes the right move is the opposite: block the crawler. If a page is gated content or something you don't want feeding a model, blocking makes sense. The trade-off is blunt: block a crawler and you also lose any chance of being cited in that engine's answers.

The bigger point: a single re-crawl is a weak thing to obsess over. What tells you whether your update landed is the citation trend over weeks. Did the corrected number start showing up in ChatGPT answers? Did your share of voice move after the edit? CiteTrack's citation tracking turns a guessing game about crawl timing into something you can actually read.

Frequently asked questions

Where to go from here

For the full guide on which AI crawlers exist and how they behave, the Crawl Academy explainer covers GPTBot, ClaudeBot, and PerplexityBot end to end. Is your website crawlable by AI covers the access layer underneath all of this, since a page a bot cannot reach never gets re-crawled at any frequency. The free AI Crawlability Checker, Crawler Log Analyzer, and Chrome extension let you verify crawl access and page readiness right now.

Guides