Two ways to tell Google not to index a page: the robots meta tag in the HTML, or the X-Robots-Tag in the HTTP response header. They do the same job. They are not interchangeable. Picking the right one comes down to what the file is (HTML vs PDF vs image), where you can edit (template vs server config), and whether you need it at scale (one page vs ten thousand).
The two mechanisms in one sentence each
robots meta tag: <meta name="robots" content="noindex, nofollow"> in the <head> of an HTML document. Works only on HTML pages because there is no equivalent in PDF / image / JSON metadata. Easy to add via the page template.
X-Robots-Tag: X-Robots-Tag: noindex, nofollow as an HTTP response header. Works on any resource type the server returns — HTML, PDF, image, video, JSON, anything. Configured in the server (Apache, Nginx, Vercel edge headers, Cloudflare Workers) or via the framework's response API.
When robots meta is the right choice
- Single-page exclusion: a specific page you want noindexed. Add the meta tag to that page's template, ship the change.
- Template-controlled pages: when the noindex decision is made per-page-type (e.g., all /thank-you/ pages should be noindexed). Add the meta tag inside the layout template that those pages use.
- Dynamic decisions: when noindex depends on runtime data (e.g., noindex pages with fewer than 200 words). The meta tag is rendered conditionally in the template.
- When you do not have server-config access: shared hosting, locked-down CMS, agency deploy where you can edit templates but not htaccess. The meta tag is your only option.
When X-Robots-Tag is the right choice
- Non-HTML resources: PDFs, images, JSON endpoints, sitemaps. You cannot add a meta tag to a PDF. X-Robots-Tag in the HTTP response is the only way.
- Pattern-based exclusion: "noindex everything under /admin/*" or "noindex every .pdf file". Configure once in the server, applies to every matching URL forever. Much cleaner than editing 200 templates.
- Large-scale exclusion: 10,000 dynamically generated URLs that should not be indexed. A single Nginx location block beats a template-level conditional.
- When you want defence in depth: combine X-Robots-Tag at the server with robots meta in the template. Either layer catches the noindex; both layers means a template bug cannot accidentally re-index the page.
What both can set
Both mechanisms accept the same directive set. The most common: noindex (do not index), nofollow (do not follow links from this page), none (= noindex, nofollow), noarchive (do not show cached version), nosnippet (do not show snippet in SERP), max-snippet:N (cap snippet length), max-image-preview:standard|large|none (control image previews in AI Overviews — relevant for 2026 GEO work), max-video-preview:N (cap video preview length), unavailable_after:date (auto-noindex after a date).
Google reads the same directives regardless of which mechanism delivers them. The mechanism choice is about scale and resource type, not directive support.
How to set X-Robots-Tag on common platforms
Vercel (vercel.json)
Use a [[headers]] block with a source pattern and a values entry for X-Robots-Tag. Example: [[headers]] source = "/admin/(.*)" [[headers.values]] X-Robots-Tag = "noindex, nofollow". Applies to every URL matching the pattern.
Netlify (netlify.toml)
[[headers]] block with a for path glob and values.X-Robots-Tag set. Example: [[headers]] for = "/admin/*" [headers.values] X-Robots-Tag = "noindex, nofollow".
Apache (.htaccess)
<FilesMatch "\.pdf$"> Header set X-Robots-Tag "noindex, nofollow" </FilesMatch> for file-pattern matching. Or use <Location /admin> Header set X-Robots-Tag "noindex, nofollow" </Location> for path matching.
Nginx
location ~ /admin/ { add_header X-Robots-Tag "noindex, nofollow"; } inside the server block. The location ~ matches by regex; use location ^~ for prefix-only matching.
Cloudflare Workers / Edge
Modify the Response headers in the worker script: response.headers.set('X-Robots-Tag', 'noindex, nofollow'). Useful for sites where the origin server cannot easily emit the header.
Verifying the header is set
curl -I https://example.com/page returns all response headers including X-Robots-Tag. Look for the X-Robots-Tag line in the output. If it is missing, the header is not being set.
For Search Console verification: the URL Inspection tool shows the directives Google picked up from both the meta tag and the HTTP header. If either layer sets noindex, Google indexes nothing. Worth checking both layers because a misconfigured CDN can strip headers silently.
The defence-in-depth pattern
The strongest production setup uses both. Server-level X-Robots-Tag for the path pattern (e.g., /admin/*) plus template-level meta robots in the page <head> for the same pages. Two layers means a template bug or a CDN misconfiguration cannot accidentally reindex pages that should be private.
This is how the admin section of this site is configured: vercel.json sets X-Robots-Tag: noindex, nofollow on /admin/*, AND the AdminLayout template emits <meta name="robots" content="noindex, nofollow"> in the head. Belt and braces.
Common mistakes
- Adding noindex to a page that is already blocked by robots.txt. Disallow in robots.txt prevents Googlebot from fetching the page at all, so it never sees the noindex directive. The page can end up indexed without snippet because Google saw the URL elsewhere but cannot fetch it. Remove the Disallow rule so the bot can fetch + see the noindex.
- Setting noindex via meta tag in JavaScript-rendered HTML. Some bots execute JS, some do not. The safe bet is server-side rendering of the noindex meta tag, OR an X-Robots-Tag in the HTTP header that does not require JS execution.
- Setting noindex on a page that has backlinks. Google will respect noindex but the link equity from those backlinks is wasted. Better to 301 the page to a related URL that absorbs the equity.
- Forgetting that X-Robots-Tag is per-response, not per-server. CDN cache hits sometimes strip custom headers; verify with curl after deploy.
The summary
robots meta tag: per-page, in the HTML, edit at the template level. X-Robots-Tag: per-response, in the HTTP header, edit at the server level. Either works for HTML; only X-Robots-Tag works for non-HTML. Use both when the noindex matters.
Adjacent reading: 410 vs 404 covers the retirement case (URLs going away); 301 vs 302 vs 307 vs 308 covers the redirect case (URLs moving). Together with noindex they are the three primary tools for telling Google what to do with a URL.