Back in 2021 I inherited a WooCommerce site with 140,000 product URLs, a single sitemap.xml that topped out at exactly 50,000 entries, and a client who couldn't figure out why Google Search Console was only showing a fraction of their pages indexed. The sitemap file was just... stopping. No error. No warning. Just silent truncation. That's the kind of thing that costs you ranking without ever sending a visible signal that something is wrong.
If your site is growing past 50,000 URLs, this post is the one you need to read before you hit that wall.
Why the 50,000 URL Limit Exists
The Sitemaps protocol, maintained jointly by Google, Bing, and others, sets two hard constraints on a single sitemap file: no more than 50,000 URLs, and no larger than 50MB uncompressed. These aren't suggestions. Googlebot will stop parsing the file at whichever limit it hits first.
Honestly, the 50MB ceiling is rarely the issue. Most URLs are short enough that you'd need comically long paths to hit the file size limit before the URL count limit. The 50,000 URL cap is the one that bites people.
So what do you do when your catalogue, your blog archive, your user-generated content section crosses that threshold? You use a sitemap index file.
What a Sitemap Index File Actually Is
Simple concept. Instead of one sitemap with URLs in it, you create a parent XML file that points to multiple child sitemaps. Each child sitemap can hold up to 50,000 URLs. The index file itself can reference up to 50,000 child sitemaps. So the theoretical ceiling is 2.5 billion URLs. You're not going to hit it.
The structure looks like this:
`` sitemap_index.xml ├── sitemap-posts-1.xml (up to 50,000 blog posts) ├── sitemap-posts-2.xml (overflow posts) ├── sitemap-products-1.xml (up to 50,000 products) ├── sitemap-products-2.xml (overflow products) └── sitemap-pages.xml (static pages) ``
The index file uses <sitemapindex> as the root element instead of <urlset>. Each child is wrapped in a <sitemap> tag with a <loc> pointing to the child file, and optionally a <lastmod> timestamp.
What you submit to Google Search Console is the index file URL, not each individual child. One submission, one source of truth.
How to Split Your URLs Intelligently
Here's where most tutorials go wrong. They say "just chunk your URLs into groups of 50,000" and leave it at that. But the chunk boundaries matter for maintainability and for Googlebot's crawl logic.
I split by content type, not by arbitrary number. Always. Every time.
Split by content type first
- Products in their own sitemap series (
sitemap-products-1.xml,sitemap-products-2.xml) - Blog posts in their own series
- Category and tag pages together (they're navigational, treat them as one group)
- Static pages (About, Contact, landing pages) in a single file, usually well under 1,000 URLs
Why does this matter? Because when a product database gets a bulk update, only the product sitemaps need regenerating. You're not invalidating and regenerating a monolithic file that mixes products with blog posts. Googlebot also tends to crawl sitemaps sequentially, so grouping by type gives it a cleaner signal about what kind of content it's going to find.
Then paginate within each type
Once a content type exceeds 50,000 URLs, paginate it. Use a consistent naming convention from day one. I use sitemap-{type}-{page}.xml . Don't use dates in the filename. I made that mistake on a news site in 2019 and ended up with filenames like sitemap-articles-2019-march.xml that became completely meaningless the moment we needed to regenerate historical content. Stick to numbered pages.
Keep lastmod honest
The <lastmod> element in your child sitemaps should reflect when the content was actually last modified, not when you regenerated the sitemap. I've seen setups where every regeneration stamps every URL with today's date. That trains Googlebot to ignore your lastmod entirely because the signal is noise. Use your CMS's actual modified timestamp. In WordPress, that's post_modified_gmt from the wp_posts table.
Tooling: What Actually Works at Scale
For WordPress sites (which is most of what we build at Seahawk), the options I actually use are:
- Yoast SEO handles sitemap indexes automatically once you have more than 1,000 URLs per post type. It generates clean, segmented files. But the default batch size is 1,000 URLs per child sitemap, which is conservative. You can filter that up with
wpseo_sitemap_entries_per_page. I usually push it to 5,000 on well-hosted servers. - Rank Math does the same, with slightly more granular control over which post types and taxonomies get their own child sitemaps. On a recent client site with 80,000 WooCommerce products, Rank Math's segmentation out of the box was cleaner than Yoast's.
- Custom generation with WP-CLI is what I reach for when the site has unusual content types or the plugin-generated sitemaps are too slow to generate. I've written scripts that query the database directly, paginate results in chunks of 10,000, write the XML to disk, and then write the index file last. Runs in under 30 seconds for 200,000 URLs. The key is writing to a temp file and atomically moving it into place so Googlebot never hits a half-written file.
For non-WordPress stacks, Screaming Frog SEO Spider can crawl your site and generate a sitemap index for you, though it's better as a validation tool than a production sitemap generator. For production, generate sitemaps from your data source, not from a crawler. Crawlers miss things.
Submitting and Validating in Search Console
Go to Google Search Console, open the Sitemaps report, and submit your index file URL. One URL. That's it.
After submission, give it 24-48 hours and then check two things:
- Submitted vs Discovered. The "Discovered URLs" count should be climbing toward your actual URL count. If it plateaus weirdly early, you likely have a crawl budget issue or a malformed child sitemap.
- Individual child sitemap status. Search Console will show you each child sitemap and its status. If one child is returning an error, you can isolate it without touching the others. This is the underrated advantage of the index structure.
Bing Webmaster Tools works the same way. Submit the index file, not the children. Takes two minutes.
Common Mistakes I See Constantly
Seahawk audits sites for agencies fairly regularly. The same sitemap errors come up again and again.
Noindexed URLs in the sitemap. If a URL has a noindex meta tag or X-Robots header, it should not be in your sitemap. Full stop. A sitemap is a recommendation to crawl and index. Including noindexed URLs wastes crawl budget and confuses the signal. I run a quick Screaming Frog crawl against the sitemap URLs and filter for noindex responses before any major site launch.
Blocked URLs in the sitemap. Even worse than noindex: URLs that are disallowed in robots.txt but still listed in the sitemap. Googlebot can see the contradiction. It won't penalise you for it, but it's sloppy and it wastes time.
Not updating the index file after adding a new child. This sounds obvious but it trips up custom implementations. You add a new content type, generate a new child sitemap file, and forget to add the <sitemap> entry to the index. The child file exists on disk but nothing points to it. I've seen this sit unnoticed for months.
Submitting child sitemaps individually instead of the index. If you have 12 child sitemaps and you submit all 12 to Search Console separately, you've lost the organisational benefit. Submit the index. Let it cascade.
Crawl Budget: The Bigger Picture
Sitemap index files connect directly to crawl budget management. Google's documentation on crawl budget is worth reading if you're operating at this scale. The short version: Googlebot has a finite amount of time it'll spend on your site per day, and a well-structured sitemap index helps it spend that time on your highest-priority content first.
I put freshly-updated content in its own child sitemap on large sites. Some implementations I've done use a sitemap-recent.xml that only contains URLs modified in the last 30 days, refreshed hourly. Googlebot recrawls that one aggressively. The older historical sitemap files get crawled less frequently, which is fine because that content isn't changing.
This isn't a trick. It's just making the crawl signal match the content reality.
FAQ
How many child sitemaps can a sitemap index file contain?
The protocol allows up to 50,000 child sitemaps in a single index file. In practice, if you're approaching that number you have hundreds of millions of URLs and you're dealing with problems most sites never face. For the vast majority of large sites, you'll have somewhere between 5 and 50 child sitemaps.
Do I need to compress my sitemap files with gzip?
You don't need to, but it's worth doing for large files. A 50MB sitemap compresses down to roughly 3-5MB with gzip. Googlebot accepts .xml.gz files without any configuration. Most modern web servers will handle this transparently if you enable gzip compression at the server level. For static-file sitemaps generated to disk, I pre-compress with gzip and serve the .xml.gz directly.
Should I include image or video sitemaps in the index?
Yes. If you're using image sitemap extensions or video sitemaps, they can be included as child sitemaps in the same index file. Keep them in their own child files rather than mixing image data into your standard URL sitemaps. It keeps the files smaller and makes it easier to regenerate just the image sitemap when your media library updates.
What happens if a child sitemap returns a 404?
Search Console will flag that child as errored in the Sitemaps report. Googlebot will still process the other child sitemaps in the index. It won't invalidate the whole index, but the URLs only listed in that 404'd child won't be discovered through the sitemap. Fix it promptly. The error lingers in Search Console's report for weeks even after you fix it, which is annoying but harmless.
Can I use a sitemap index on a small site?
Technically yes, but there's no reason to. The added complexity isn't worth it below 10,000 URLs. A single sitemap.xml is simpler to maintain, simpler to debug, and does exactly the same job. Use an index file when you need it, not before.
---
The 50,000 URL limit catches people off guard every time, usually at the worst possible moment, like right before a product launch. Getting the index structure right once means you never have to think about it again as the site grows. That's worth an hour of setup.