A client rang me in 2022 — a UK-based e-commerce operator with roughly 14,000 product pages — furious that they'd dropped 34% of their organic traffic in six weeks. No manual penalty. No algorithm announcement. Just a slow, quiet collapse. We ran a full crawl with Screaming Frog and found the problem inside 90 minutes: their pagination had been auto-generating thousands of near-duplicate URLs, Google had crawled all of them instead of the real product pages, and their crawl budget was completely gone. Wasted. Every month.
That's the thing about large-site SEO. The problems aren't harder to understand — they're just catastrophically larger in consequence. A canonical tag misconfigured on a 20-page site is annoying. On a 14,000-page site, it can quietly strangle your whole index.
This is the audit checklist I use atSeahawk Mediawhen a site crosses the 10,000-page mark. In no particular order of importance — because every large site has its own hierarchy of disasters.
---
Start With Crawl Budget — Not Keywords
Most people open a large-site audit by looking at rankings. Wrong order. Completely. Rankings are downstream of indexation, and indexation is downstream of crawl budget. Fix the order of operations.
Crawl budget, for anyone who needs the plain version: it's the number of URLs Googlebot will crawl on your site within a given timeframe.Google's own documentation on crawl budgetis genuinely worth reading here — they're quite specific about what wastes it.
What's burning your budget?
Pull your server logs first. Not GSC data — actual server logs. I useGoAccessfor quick analysis on large log files because it handles volume without crying. What you're looking for:
- Faceted navigation URLs (e.g.,
/shoes?colour=red&size=10&sort=price) - Session IDs appended to URLs
- Infinite scroll or "load more" implementations generating unique parameter strings
- Duplicate paginated URLs (
/page/1and/) both being crawled - Internal search result pages that aren't blocked
Any site over 10,000 pages with an active faceted navigation is almost certainly haemorrhaging crawl budget. Almost certainly. The fix isn't glamorous — it's a robots.txt disallow on the parameter patterns, or ideally,proper URL parameter handling via GSCcombined with canonical tags on the faceted pages themselves.
Back in early 2021, Seahawk had a furniture retailer client with 23,000 product URLs. Looked fine on the surface. But their log analysis showed Googlebot spending 61% of its crawl visits on faceted filter combinations that had zero search demand and zero unique content. Their real product pages were getting crawled roughly once every 14 days. Switched the facet parameters tonoindex, followand disallowed the heavy combinatorial patterns in robots.txt. Within six weeks, average crawl frequency on real product pages dropped to every 3–4 days.
---
Indexation Audit: What's Actually in Google's Index?
site:yourdomain.comin Google gives you a rough figure. Don't rely on it for precision, but it's a quick sanity check. Cross-reference with GSC's Index Coverage report.
The gap between "pages you want indexed" and "pages Google has indexed" is where the money is. On large sites, this gap tends to be enormous and entirely preventable.
The four states you care about
- Indexed, no issues— fine, leave it
- Excluded: noindex— intentional? Confirm it is
- Excluded: crawled, currently not indexed— this is the one that should alarm you
- Excluded: discovered, not crawled— crawl budget problem, come back up to section one
"Crawled, currently not indexed" is Google's way of saying:I got here, I looked around, and I decided not to bother.That usually means thin content, near-duplicate content, or a quality signal so weak Google is making an active choice to skip it. On product pages, this often happens with auto-generated descriptions that are three sentences of boilerplate. Google has seen a thousand versions of "This product is available in multiple colours and ships within 3–5 working days." It doesn't want another one.
---
Canonical Tags at Scale
Canonicals are where I see the most spectacular self-inflicted damage on large sites. Not because they're complicated — they're not — but because at 10,000+ pages, a single template error propagates instantly across thousands of URLs.
The two failures I see constantly:
Self-referencing canonicals that don't actually point to the right place.Classic example: a paginated category page wherepage/2has a canonical pointing to itself instead ofpage/1or the root category. Multiply that by 400 category pages with 8 pages of pagination each and you've got 2,800+ pages with broken canonical signals.
Canonical chains.Page A canonicalises to Page B, which canonicalises to Page C. Google follows canonical chains, but it's not enthusiastic about them. Three hops is already pushing it. I've seen sites with five-hop chains built up over years of migrations and redesigns. Screaming Frog's "Canonical" tab will show you this directly — export it, filter for chains.
Run a full canonical audit on every template type separately. Product pages. Category pages. Blog posts. Tag archives. Author pages. Each template has its own failure mode, and you won't catch them all from a random sample.
---
XML Sitemaps: More Important Than People Think
At 10,000+ pages, a single sitemap file starts becoming a problem. Google's limit is 50,000 URLs or 50MB per sitemap file — but hitting that limit isn't the point. The point is that a monolithic sitemap with 40,000 URLs is hard to monitor and hard to debug when things go wrong.
Break it up. Use a sitemap index file pointing to segmented sitemaps:
- Products sitemap
- Categories sitemap
- Blog/editorial sitemap
- Brand or manufacturer pages sitemap (if applicable)
Why does segmentation matter? Because when something breaks — and it will — you can isolate the problem. If Google is suddenly not picking up your new product pages, you check the products sitemap crawl date in GSC and debug from there. A monolithic sitemap gives you nowhere to look.
Also:only include URLs you actually want indexed in your sitemap.This sounds obvious. You'd be surprised. I've audited sites where the sitemap was auto-generated by a plugin and included tag pages, author archives, attachment pages, and half-a-dozen other URL types that hadnoindexon them. Pointless noise.
Validate your sitemap withGoogle's Rich Results Testif you're also dealing with structured data — and check raw sitemap delivery in a browser to confirm your server is returning a200, not a301chain or, god forbid, a404.
---
Internal Linking at Scale: The Underrated One
PageRank is still real. It flows through internal links. On a large site, the architecture of your internal linking effectively decides which pages have authority and which are orphans dying quietly in a corner.
Seahawk had a publishing client in 2023 — roughly 18,000 articles across a news and lifestyle vertical. Their top-funnel category pages were getting decent traffic. But their deeper archival content — stuff from 2015 to 2019 that still had genuine search demand — was nearly invisible. Not because the content was bad. Because nothing linked to it anymore. They'd redesigned their category navigation three times, and each time, older content got buried one more level deep.
The fix was unglamorous: we built a programmatic internal linking strategy using a custom WordPress plugin that identified articles with relevant keyword overlap and inserted contextual links. Click depth on their archival content dropped from an average of 7.2 clicks from the homepage to 3.1. Organic impressions on those pages rose 28% over the following quarter.
Here's a quick internal linking checklist for large sites:
- No page you want indexed should be more than 3 clicks from the homepage
- Orphan pages (zero internal links pointing to them) should be treated as an emergency, not a backlog item
- Breadcrumb navigation counts as internal linking — make sure it's implemented properly and uses real anchor text, not just "Category > Subcategory" with generic labels
- Check for pages with only one internal link pointing to them — that's barely better than orphaned
---
Structured Data and Schema at Scale
If you've got 10,000+ product pages and none of them haveProduct schemawithOffer,Review, andAggregateRatingproperties, you're leaving SERP real estate on the table.
But structured data at scale also introduces its own audit requirements. A schema error in a template means thousands of invalid markup instances. I check structured data with two tools in combination: Google's Rich Results Test for individual URL sampling, and a crawl-level schema extraction in Screaming Frog (Configuration → Custom Extraction → XPath for JSON-LD blocks) to get a bulk view across all page types.
What to look for:
- Missing required properties (especially
priceandpriceCurrencyon Product pages — these are common omissions) - Mismatched structured data (schema says one product name, the
<title>says another) - Deprecated schema types —
DataFeedElementand some olderitemscopemicrodata patterns are worth auditing out - Review schema that violatesGoogle's review snippet guidelines— first-party reviews marked up as third-party, or aggregated scores from tiny sample sizes
---
Page Speed at Scale: Don't Audit What You Can't Fix
Core Web Vitals matter. But here's the thing that doesn't get said enough: auditing CWV across 10,000 pages and trying to fix every individual URL is a fool's errand. You audit by template, then fix by template.
Run a sample — 20–30 URLs per template type — through PageSpeed Insights orWebPageTest. If your product pages have an average LCP of 4.8s, that's a template-level problem. The fix is in your image delivery pipeline, your critical CSS, or your server response time — not in touching individual pages.
On large WordPress sites specifically (which is most of what we work with at Seahawk), the usual culprits at scale are:
- Unoptimised WooCommerce product images served without WebP conversion
- Too many HTTP requests from poorly-scoped plugin enqueues on pages that don't need those scripts
- Hosting tiers that haven't scaled with site growth — a plan that was fine at 2,000 products is often drowning at 12,000
Get your hosting right first. Everything else is decoration.
---
Redirect Audit: The Migration Debt Problem
Large sites accumulate redirect chains the way old houses accumulate dodgy wiring. Each redesign, each domain migration, each URL restructure adds another layer. After four or five years, it's not uncommon to find redirect chains four or five hops deep.
Every hop costs time. Every hop dilutes the PageRank signal being passed. And some very old 302s that were meant to be temporary are still sitting there doing very permanent damage.
My process:
- Crawl with Screaming Frog, export all 3xx responses
- Filter for chains (A → B → C, or longer)
- Update all source links to point directly to the final destination
- Confirm the final destination is a 200, not another redirect
- Flag any 302s that should be 301s and get them changed at the server level
Also check: are any of your XML sitemap URLs returning redirects? Because that's a common one. A sitemap should only contain URLs that return 200s. If your sitemap is full of 301s, you're doing Google's job for it and doing it badly.
---
FAQ
How long does a technical SEO audit take for a 10,000+ page site?
Honestly, it depends on how well-instrumented the site is. If they've got GSC set up properly, server logs accessible, and Screaming Frog can crawl without rate-limiting itself, a thorough audit takes me about 3–5 working days for the data collection and analysis phase alone. Reporting is another 1–2 days. Anyone telling you they can do a meaningful large-site audit in an afternoon is sampling, not auditing.
Do I need to audit every single page or can I work from samples?
Work from templates, not individual pages. A site with 12,000 product pages has maybe 4–6 meaningful page templates. Audit each template type thoroughly with a representative sample (20–30 URLs minimum), and your findings will apply across the entire template. The exception is orphan page identification and redirect chain discovery — those need full crawl coverage, not sampling.
What's the single highest-impact fix on most large sites?
Crawl budget, nine times out of ten. Specifically, blocking or canonicalising faceted navigation URLs that have no search demand and no unique content. I've seen this single fix move the needle more than any other change on e-commerce sites with large catalogues. It's unglamorous work — robots.txt edits, canonical tags, parameter configurations — but it often produces faster results than any content or link-building effort would.
Should I use Screaming Frog or Sitebulb for large sites?
Both are good. I use Screaming Frog for the majority of my crawl work because I know its export formats inside out after years of use, and its custom extraction options are excellent. Sitebulb has a genuinely better visualisation layer and its audit report is more readable for clients. For sites over 50,000 pages, you might also look atDeepCrawl (now Lumar)for cloud-based crawling that doesn't depend on your local machine's RAM.
What's the most commonly missed issue on large-site audits?
Internal linking depth. Everyone checks for broken links and canonicals. Very few people systematically identify pages that are six or seven clicks from the homepage and ask why they're expected to rank for anything competitive. Click depth is a proxy for crawl priority and authority distribution. Audit it every time.
---
Large-site SEO isn't a different discipline — it's the same principles at a scale where the consequences of neglect compound fast. The checklist above won't stay static. Every site has its own particular chaos. But if you work through crawl budget, indexation, canonicals, sitemaps, internal linking, structured data, page speed, and redirects in that rough order — you'll find 80% of what's broken before you've looked at a single keyword.
Start with the infrastructure. The rankings follow.