The sitemap Google refused to read for six months, and the 14-second fix

If Google Search Console shows your sitemap index as Success while Discovered pages sits at zero, and the child sitemaps stay on "Couldn't fetch" no matter how many times you resubmit them, you are probably not looking at a server problem. You are looking at a per-URL backoff inside Google's sitemap scheduler. The fastest way to prove it is to resubmit the identical file under a slightly different URL and watch what happens.

That diagnosis took me six months, five separate audits and four different AI models. The test that finally settled it took 14 seconds.

Key takeaway: A sitemap index can report Success while every child sitemap in it is invisible to Google. If your edge logs show no Googlebot request at all for a "Couldn't fetch" child, the failure is on Google's side, not yours. Resubmitting the same file at a versioned URL such as /sitemap/0.xml?v=20260815 bypasses the backoff and processes in seconds.

What "Success" was actually reporting

The site is Deluxe Astrology, my parents' Vedic astrology platform and the largest thing I run: roughly 240,000 URLs across 30 languages, published from Supabase and served as a sitemap index with 63 child sitemaps under it. I write about the architecture behind it on the Deluxe Astrology project page.

Indexed pages had been sliding for weeks. Impressions fell off a cliff in late June. And the one system whose entire job is to hand Google a list of fresh URLs was showing a green tick.

Open the sitemap index in Search Console and it says "Sitemap index processed successfully". Click into it and the Sitemaps read table says 0-0 of 0. Google had fetched the index three separate times over three weeks, parsed it correctly as an index, and registered none of its children. Not one child sitemap had ever been fetched. When I submitted three of them directly, they sat on "Couldn't fetch" for hours.

This is the trap. The status on a sitemap index is a statement about the index file only. It tells you the XML parsed and the child URLs looked well formed. It tells you nothing about whether Google ever went and read them. That distinction is buried in a table most people never scroll to, and it is the whole ballgame on any site big enough to need a sitemap index in the first place.

Everything on the server checked out, five separate times

If you have hit "Couldn't fetch" you know how the next hour goes. You curl the URL. You check robots.txt. You check the firewall. You check the CDN cache. You run the live test in URL Inspection. Everything comes back clean. So you tell yourself the classic Search Console lie: it just needs time, check back in 24 to 72 hours.

Over the months this became a genuinely thorough investigation, and not just by me. I ran the problem through several frontier models as independent auditors: Claude, Kimi, GLM and a couple of custom agents, each with access to the codebase, the Search Console data and the edge logs. I have written before about how many AI models are actually worth running at once, and this was the case that tested it hardest.

They agreed on almost everything, and they were right about almost everything:

• The live sitemap index served a valid <sitemapindex> with all 63 children, correct content type, no byte order mark, no cross-host URLs.
• Every child served a valid <urlset> with the expected URL counts.
• Googlebot's own live fetch of the index and of a child, through URL Inspection, returned the real XML. "URL is available to Google."
• The edge logs showed real Googlebot requests to the index being allowed and served 200 from cache.
• Nothing in robots.txt, middleware, redirects or CDN config touched the sitemap paths.

One earlier mistake was real and had already been fixed. A per-IP rate limit on the WAF, added months before to fight a bot farm that was setting fire to my hosting bill, had for a while been challenging crawler traffic. That rule was corrected. After the correction, every audit reached the same conclusion: the server side is clean, Google needs time, check back in 24 to 72 hours.

Five audits. Same verdict. Same recommendation. The numbers never moved.

The tell was an absence, not an error

The uncomfortable fact was hiding in the edge logs, in what was not there. After submitting three child sitemaps, there was no Googlebot request for any of them. Not an allowed one, not a challenged one, not a denied one. Google was not failing to fetch the children. Google was choosing not to try.

You cannot diagnose that from the server, by construction. Nothing arrives to inspect. Every tool in the standard kit is built to explain a request that went wrong, and there was no request. This is the one class of problem that log file analysis solves by omission rather than by evidence: you go looking for the hits, and the answer is the empty result set.

Search Console's live test makes it worse, because it bypasses whatever scheduler is making that choice. It fetches on demand, from a different code path, and cheerfully reports "available" for a URL the sitemap subsystem will not queue. A green live test is not proof that the sitemap pipeline will ever touch the file.

The API told the truth the UI would not

The Search Console Sitemaps API gave the first honest signal. The UI says "Couldn't fetch", which reads as a failure with a cause. The API returns isPending: true with errors: 0 and no lastDownloaded timestamp at all.

Those are different claims. "Couldn't fetch" implies an attempt that failed. isPending with zero errors means no attempt was ever made. Six months of debugging had been aimed at a failure that did not exist.

If you run anything at scale, get the API wired up before you need it. It is one OAuth scope and a few lines of code, and it is the difference between a status string designed for reassurance and the actual state of the record. I lean on it heavily in my Claude Code SEO audit workflow for exactly this reason.

The controlled experiment

Rather than run a sixth audit, I ran a control.

First, a baseline. I submitted a sitemap Google had never seen before, a small one for a side section of the site. It was downloaded and processed 34 seconds after submission. So the pipeline was healthy, the host was reachable, and Google was willing to fetch sitemaps from this domain right now.

Then the real test. I resubmitted one of the stuck children, byte for byte identical, served by the same route on the same server, with exactly one difference: a query string on the end. /sitemap/0.xml?v=20260815.

Same file. Same server. Same bytes. Different string. One was invisible for two and a half hours and counting, the other was read in 14 seconds.

That is the entire diagnosis, and it is the reason I keep arguing that a controlled test beats another round of verification. Every audit had confirmed facts about the server. Not one had varied the only input that turned out to matter.

What was actually happening

Google's sitemap system was holding those exact 63 URLs in a per-URL failure backoff.

Months earlier, every read of the index had triggered a burst of 63 child fetches from a single Google IP within seconds. That is normal Googlebot behaviour for an index: it reads the parent, then goes and gets the children more or less at once. The per-IP rate limit on the WAF, sized for average human traffic, saw a burst of 63 requests from one address and did what it was configured to do. It challenged them. Every cycle. For weeks.

The single index request itself always squeaked through, because one request is not a burst. That is why the index kept reading "Success" while its children never came into existence. The rule was perfectly shaped to break exactly the URLs I most needed Google to read, while leaving the one URL that reports on them untouched.

Fixing the firewall did not clear Google's memory of the URLs that had failed. It only stopped creating new failures. The backoff on those 63 specific strings survived the fix, and nothing about waiting was going to expire it on a timescale I could observe.

The fix: 63 submissions, zero deploys

Through the Search Console API I resubmitted all 63 children with the version query string. Within about two minutes every one of them had been fetched and processed: 155,545 URLs registered with Google, zero errors, zero warnings. Discovered pages, which had read zero for half a year, populated while I watched.

The durable fix is a two-line change in the next release: have the sitemap index emit the versioned child URLs, so the index and robots.txt point at URLs Google is willing to read. Bump the version token whenever the generation logic changes and you get a clean cache-busting mechanism for free.

One honest caveat. Indexing is not discovery. Google is crawling this host at a trickle after months of distrust, and 155,000 URLs do not get swept in a week. Discovery working again is the precondition, not the outcome. If your crawl budget is already thin, fixing the sitemap is where the work starts.

Why five audits converged on the wrong answer

This is the part I keep coming back to.

The models were excellent at verification. Given a claim about content type, cache headers, robots directives or middleware, they checked it accurately and reported it honestly. What none of them did, on their own, was propose submitting the same file under a different name to see what happened.

Convergence on a shared blind spot looks exactly like consensus. Five auditors reading the same evidence with the same assumption, that a fetch failure implies a fetch attempt, will produce five confident agreements and zero progress. The agreement feels like confirmation. It is actually correlation between the auditors, not between the auditors and reality.

What broke the deadlock was not another audit. It was deciding that six months of "wait 72 hours" was a hypothesis rather than a plan, and designing a test where the only variable was the URL string. That is a human job, and I do not think it stops being one soon.

The five rules I took away

• Success on a sitemap index is about the index file, not the children. Scroll to "Sitemaps read". If it says zero, the green tick is decorative.
• "Couldn't fetch" in the UI means "not processed yet" in the API. Read the API. isPending and lastDownloaded are what you actually want to know, and the UI shows you neither.
• Google remembers your infrastructure mistakes longer than you do. A rate limit that challenged the crawler for a few weeks can leave specific URLs in backoff long after the rule is gone. Waiting does not reliably clear it. Versioning the URL does.
• Size rate limits for crawler bursts, not average traffic. An index read triggers N child fetches from one IP within seconds. Any per-IP limit below N will challenge exactly the URLs you most need Google to read, while the index itself passes and reports success.
• When several models agree that the server is clean and you should wait, they are probably right about the server and wrong about the wait. Run a control instead of a sixth audit.

If you think you have the same problem

Work through this in order. It takes about fifteen minutes and it separates a server problem from a Google-side backoff cleanly.

• Open the sitemap index in Search Console and read the Sitemaps read count, not the status. Zero children read on a healthy index is the signature.
• Pull the same sitemaps through the Search Console API. Note isPending, errors and lastDownloaded for each one.
• Search your edge or CDN logs for Googlebot requests to the specific child paths over the last 30 days. No requests at all, of any status, means the problem is not on your server.
• Submit a brand new sitemap URL Google has never seen. If it processes in under a minute, your host and your pipeline are fine.
• Resubmit one stuck child with a version query string. If that processes and the plain URL does not, you have your answer and the remedy in the same step.
• Audit your WAF rate limits against burst behaviour, not averages, so the backoff does not rebuild itself. Then check the rest of your indexation fundamentals on large sites.

For the reference material behind all of this, Google's own guide to building and submitting a sitemap and the sitemaps.org protocol are still the only two documents that matter. Neither mentions per-URL backoff, which is part of why this took so long.

If you run a site past 100,000 URLs and you are stuck on the same symptom, this is the kind of thing I do for a living: technical SEO on large sites, including the programmatic SEO builds where sitemaps stop being a formality and start being the whole distribution channel. I have the API scripts, the diagnostic sequence and the scar tissue.

FAQ

Why does my sitemap index say Success but show zero discovered pages?

Because the status refers only to the index file. Google parsed your <sitemapindex> and found well-formed child URLs, which is all "Success" claims. Whether it then fetched those children is reported separately in the Sitemaps read table. Zero read on a valid index means the children were never processed, and the green tick is telling you nothing useful.

What does "Couldn't fetch" actually mean in Search Console?

Less than it sounds like. In the Search Console API the same sitemap usually returns isPending: true with errors: 0 and no lastDownloaded value, which means Google has not attempted the fetch yet rather than attempted and failed. Check the API before you spend time debugging a failure that may never have occurred.

How long should I wait before assuming a sitemap is genuinely stuck?

A sitemap Google is willing to read typically processes in seconds to minutes, not days. If a child sitemap has been pending for more than about 24 hours while a brand new sitemap URL on the same host processes immediately, waiting longer is not a strategy. Run the versioned resubmission test instead.

Does adding a query string to a sitemap URL cause duplicate content or other SEO problems?

No. A sitemap is a discovery file, not an indexable page, and the URLs inside it are unchanged. Google treats /sitemap/0.xml and /sitemap/0.xml?v=20260815 as two distinct sitemap resources, which is precisely the property you are exploiting. Point your index and robots.txt at the versioned URLs so there is one canonical set in play.

Can a WAF rate limit break sitemap discovery without breaking anything else?

Yes, and that is what makes it so hard to spot. Reading a sitemap index triggers a burst of child fetches from a single Google IP within seconds, so a per-IP limit sized for human traffic will challenge the children while the single index request passes. Everything else on the site, including live URL Inspection tests, keeps working perfectly.

Will fixing the sitemap immediately restore lost impressions?

No. Discovery and indexing are separate stages. Getting 155,000 URLs registered restores the input to the pipeline, but crawl rate on a host that has been failing for months recovers gradually, and indexing decisions follow crawling. Expect weeks, and use the time to make sure the pages being discovered are worth indexing.