A client rang me in October last year, proper panicked. His hosting dashboard showed a 34% spike in crawl traffic over three weeks and he was convinced he'd been hit by a scraper attack. I pulled up his server logs. Not a scraper. ClaudeBot and GPTBot, crawling his 800-page WooCommerce catalogue every few days like clockwork. He had no idea these things even existed.
That conversation is happening everywhere right now. And the advice people are getting online is all over the place: block everything, allow everything, it doesn't matter, it matters enormously. Most of it is written by people who haven't actually looked at a server log in months.
I have. Across 12,000+ sites built at Seahawk Media, plus my own client portfolio, I've now got a fairly clear picture of what's actually happening and what the right call is for different types of sites. Let me walk through it.
What ClaudeBot and GPTBot Actually Are
Quick grounding first, because people conflate these.
GPTBot is OpenAI's crawler. It's used to collect training data for future GPT models, and it's been active since mid-2023. OpenAI published their GPTBot documentation and stated clearly that blocking it won't affect how ChatGPT answers questions about your existing content, it's purely for future training.
ClaudeBot is Anthropic's equivalent, used to train Claude models. Same basic story. It shows up in your logs with a ClaudeBot user-agent string and identifies itself honestly, which I'll give them credit for.
There are others too. Google's Google-Extended is specifically for Gemini training, separate from Googlebot. CCBot (Common Crawl) has been around for years. Bytespider from ByteDance. The list is growing.
Here's the thing though: not all of these behave identically. GPTBot and ClaudeBot do respect your robots.txt. Bytespider... less reliably, in my experience.
The Blocking Mechanism: robots.txt and What It Actually Does
If you want to block GPTBot, you add this to your robots.txt:
`` User-agent: GPTBot Disallow: / ``
Same pattern for ClaudeBot, Google-Extended, and so on. Simple. Takes thirty seconds in any WordPress plugin like Yoast or Rank Math, or you edit the file directly.
But here's what that actually means: it's a polite request, not a lock. Legitimate crawlers from companies with legal and reputational skin in the game (OpenAI, Anthropic, Google) do respect it. Dodgy scrapers building competing tools? They ignore it completely. So if your concern is preventing data theft by bad actors, robots.txt is not your line of defence. You'd want Cloudflare's bot management or server-level IP blocking for that.
I've seen site owners spend two hours configuring robots.txt rules to block some no-name AI crawler that was absolutely never going to pay attention. Waste of time. Focus your energy on the legitimate crawlers, because that's where your actual choices live.
The Real Trade-off Nobody Talks About
Alright. Here's where most of the discussion falls flat.
The argument for blocking goes like this: "They're taking my content to train their models and I get nothing." Fair. Genuinely. If you're a publisher with original editorial content, a legal database, a library of in-depth tutorials, that's your intellectual property and the value exchange is murky at best.
The argument for allowing goes like this: "Being cited in AI answers is the new backlink. You want to be in the training data." Also somewhat fair. But shakier than people admit, because the relationship between training data and AI citation is not that direct.
Here's my actual take, after watching this play out across client sites for over a year.
The question isn't "block or allow" globally. It's "which content, for which crawlers, and what do I actually need to protect?"
A SaaS blog exists to generate awareness. Block GPTBot from that? You're cutting off a potential distribution channel for zero tangible benefit. A paid recipe database or a premium research archive? The calculus flips completely.
Back in early 2024 I was working with a client who runs a subscription-based industry report service. Decent traffic, around 40,000 monthly visitors, most of it paywalled. They were fully allowing all AI crawlers because "more visibility." I looked at what was actually crawlable by bots and it included 18 months of full-text premium reports that Google had been told to noindex. We blocked the AI crawlers from those directories. Took twenty minutes. The public marketing pages stayed open.
That's the nuance that's missing from the flat "block everything" or "allow everything" discourse.
How This Affects SEO (And How It Doesn't)
Let me be direct: blocking ClaudeBot and GPTBot has zero effect on your Google rankings. None. Googlebot is a completely separate system. GPTBot and ClaudeBot do not feed into Google Search. Don't let anyone tell you otherwise.
What it potentially affects is your visibility inside AI-generated answers. ChatGPT, Claude, Perplexity. Perplexity in particular does live web retrieval, so it's a different mechanism from training data anyway.
The Emerging AEO Angle
Answer Engine Optimisation is a real thing people are now tracking. Being cited as a source in AI responses matters more than it did eighteen months ago, though it's still hard to measure cleanly. The theory is that content in training data helps models develop familiarity with your brand or site, which could influence citations over time. It's plausible. It's not proven.
I wouldn't make a blocking decision purely based on this speculation. Make it based on the nature and commercial sensitivity of your content.
Server Load: A Legitimate Concern for Smaller Hosts
This is the boring practical one that gets overlooked in the philosophy debates.
ClaudeBot and GPTBot can be aggressive crawlers on sites they find interesting. That 800-page WooCommerce client I mentioned? His shared hosting was genuinely feeling it. We moved him to a crawl rate limit via Cloudflare's bot fight mode rather than outright blocking, which smoothed it out without cutting them off entirely.
If you're on a cheap shared host with a large site, check your logs. Actually look. You can do this in cPanel under "Raw Access" or via a tool like GoAccess, which is free and brilliant for parsing access logs quickly. If AI crawlers are appearing in the top 10 requesters and you're on limited resources, that's a real conversation to have.
On a VPS or dedicated server with sensible specs? Probably not an issue. But don't assume.
Platform-Specific Notes
WordPress Sites
Yoast SEO and Rank Math both let you edit robots.txt without touching files directly. Yoast's interface under SEO > Tools > File Editor is fine for this. Rank Math has a similar path under General Settings > Edit robots.txt.
If you want more granular control, the Crawl Optimization settings in Yoast Premium let you reduce unnecessary crawl surface generally, which helps across all bots.
Shopify and Hosted Platforms
Shopify auto-generates your robots.txt.liquid and it's been updated to include some AI crawler rules, but you do have limited control. There's a robots.txt.liquid customisation path available since 2021, but it requires theme editing. Not always obvious.
Static Sites (Gatsby, Astro, Next.js)
Full control. Just maintain your robots.txt as a static file in your public directory and you're done. No excuses for not having this configured intentionally.
My Current Recommendation Framework
Different sites warrant different defaults. Here's how I think about it now:
Allow by default if:
- Your content is primarily marketing, educational, or awareness-focused with no paywalled value.
- You're a small business trying to increase brand presence wherever possible.
- You publish freely available information and would genuinely benefit from wider distribution.
- You don't have the technical resources to maintain a nuanced blocking strategy.
Block or restrict if:
- You have premium paywalled content that is technically crawlable (common oversight).
- You run a publisher, news organisation, or database where content is the product.
- Server load from crawlers is measurably impacting performance on limited hosting.
- You have legal reasons to restrict data collection (medical, legal, financial content in regulated contexts).
Middle path worth considering:
- Block AI training crawlers from high-value content directories, allow them on blog/marketing sections.
- Use Cloudflare rate limiting rather than full blocks if load is the concern.
- Revisit quarterly. This is moving fast and what makes sense today may shift.
How to Actually Check If They're Crawling You Now
Don't guess. Look.
- Cloudflare Analytics (free tier) shows bot traffic categories and you can filter by user-agent.
- GoAccess run against your raw access logs is the fastest way to see exactly which bots are hitting what.
- Google Search Console won't show you AI crawler data, but it gives you a crawl baseline for comparison.
- Search your
access.logfile for "GPTBot" or "ClaudeBot" directly:grep -i "GPTBot" /var/log/nginx/access.log | wc -lwill tell you how many requests in a given log file quickly.
Seahawk had a project earlier this year where the client was certain AI crawlers were the source of their performance issues. Ran the log analysis. GPTBot had made 47 requests in 30 days. Completely irrelevant. The actual culprit was a misconfigured backup plugin running full site scans every 6 hours. Don't let the bot discourse distract you from your actual logs.
FAQ
Does blocking GPTBot hurt my ChatGPT visibility?
For content already indexed and in training data, no. Blocking GPTBot now only affects future training runs. ChatGPT's existing knowledge won't change because you added a robots.txt rule today. And ChatGPT's browsing plugin does live web retrieval anyway, which is governed by different mechanisms.
If I allow ClaudeBot, will Anthropic credit my site?
Not automatically, no. Training data consumption doesn't come with attribution. If Claude cites your site in a response, that's based on how it was trained and what it retrieves in real-time, not a direct contractual result of you allowing the crawler.
Can I block just specific directories from AI crawlers?
Yes, absolutely. Your robots.txt can be as granular as you like. Disallow: /premium/ for GPTBot while leaving /blog/ open is perfectly valid syntax. This is what I'd actually recommend for most content businesses.
Is there a legal basis for demanding AI companies don't train on my content?
This is genuinely unsettled territory. The New York Times lawsuit against OpenAI is the most prominent case working through this. For now, robots.txt is the practical mechanism available. Legal frameworks are years behind the technology.
Do all AI crawlers respect robots.txt?
The major ones from well-funded, reputationally exposed companies (OpenAI, Anthropic, Google) generally do. Smaller, less accountable scrapers often don't. robots.txt is a gentleman's agreement, not a technical barrier.
---
Look, there's no universal right answer here. I've blocked AI crawlers for some clients and explicitly opened the doors for others, sometimes on the same day. The decision belongs in the same conversation as your content strategy and your hosting setup, not in a blanket policy based on whatever you read on Twitter last week.
Check your logs. Know what's crawling you. Then make an intentional choice rather than inheriting a default you never set.