A client rang me last October, panicked. His e-commerce site was getting a tonne of traffic from what his analytics showed as "direct" visits, but conversions had dropped. After digging into the server logs for about an hour, I realised a significant chunk of that traffic wasn't humans at all. It was AI agents, specifically shopping assistants and LLM-powered comparison tools, crawling his product pages and bouncing because they couldn't parse the content properly. The structured data was a mess. Prices were buried in JavaScript. Product names lived in <h3> tags styled to look like <h2> s. The agents were arriving, finding noise, and leaving.
That's where we are now. AI agents are not coming. They're already here, already reading your sites, already making decisions based on what they can or can't extract. And most of the 12,000-plus sites I've built or touched over the years at Seahawk were not built with this in mind. Neither were yours, probably.
What "Agent-Readable" Actually Means
Let me be blunt about something. Agent-readable doesn't mean "AI-friendly" in the vague marketing sense. It means a machine can arrive at a URL, parse the content without executing three layers of JavaScript, extract the thing it's looking for, and act on it. That's it.
Humans are forgiving. We scan. We infer. We see a big number on a pricing page and we know it's the price even if it's wrapped in a <div class="fancy-number">. An AI agent following a structured workflow? Not so forgiving. It needs signals, hierarchy, and predictability.
The agents doing this work right now include things like OpenAI's browsing tool, Perplexity's online mode, and dozens of custom agents being built with frameworks like LangChain and AutoGen. They all share a common need: clean, parseable, semantically meaningful HTML with supporting structured data.
The Real Problem: JavaScript-First Architecture
This is the one I see most often. Agencies and freelancers build sites on React or Next.js (or Vue, or Svelte, fine), they do client-side rendering, and they consider the job done. The site looks beautiful. Google can crawl it, more or less, because Googlebot has a headless Chrome renderer baked in now.
But most AI agents are not running headless Chrome. They're fetching raw HTML via HTTP. If your content only exists after JavaScript executes, those agents get a blank page or a loading spinner serialised as text.
Back in 2022 Seahawk had a fintech client who'd built their entire rates and product comparison section in a React SPA. No SSR, no static fallback. We ran a simple curl against their URL and got literally 14 lines of HTML: a <div id="root"> and some script tags. That's what an agent sees. Fourteen lines.
The fix isn't necessarily abandoning your JS framework. It's server-side rendering (SSR) or static site generation (SSG). Next.js with getServerSideProps or getStaticProps. Nuxt for Vue. SvelteKit. Even just pre-rendering key pages with something like Prerender.io if you're stuck on a legacy setup. The content needs to be in the initial HTML payload.
Semantic HTML Is Not Optional Anymore
I know. You've heard "use semantic HTML" since 2009. But I'm not saying it for SEO reasons right now. I'm saying it because agents use semantic tags to understand what kind of thing they're looking at.
<article>, <nav>, <main>, <aside>, <header>, <footer>, these aren't decorative. They're signals. An agent trying to extract the main content of a page will look for <main> first. If you've built your layout with nested <div> s and nothing else, the agent has to guess. Agents that guess badly return wrong answers to users.
Here's what I audit in every site now:
- Is there exactly one
<main>element per page? - Do headings follow a logical hierarchy (
<h1>to<h2>to<h3>) without skipping levels? - Are navigation menus in
<nav>elements? - Is supplementary content (sidebars, related posts) in
<aside>? - Are lists of items actually
<ul>or<ol>, not a string of<div class="item">s?
These feel like basic things. And yet I'd say 60% of the WordPress sites I review fail at least three of them. The heading hierarchy one is almost universal, designers style <h3> s to look big and <h2> s to look small and nobody fixes the markup underneath.
Schema Markup: Do the Actual Work Here
This is where most guides go generic. I'll try not to.
Schema.org structured data tells agents not just what's on the page, but what type of thing it is. A product. A recipe. A local business. An FAQ. An event. And it gives that data in a format agents can consume without parsing prose.
The types that matter most right now, in my experience:
- Product (with
offers,price,availability, all three, always) - LocalBusiness (with
openingHoursSpecificationandgeocoordinates, not just an address string) - Article (with
datePublished,dateModified, andauthoras aPersontype, not a plain string) - FAQPage (more on this below)
- BreadcrumbList (underrated, gives agents a map of your site hierarchy)
- HowTo (if you publish tutorials or process documentation)
For WordPress sites I use Yoast SEO or Rank Math for the basics, then manually extend with custom schema in a <script type="application/ld+json"> block for anything they don't cover. The JSON-LD format is the one to use. Not RDFa, not microdata. JSON-LD. It keeps the markup clean and agents can grab it without touching your presentation layer.
One mistake I made in 2021: I put schema on a client's product pages but left the price field empty because their prices were "contact for quote." The schema validator passed it. But agents were extracting a blank price and returning that to users as "price: unknown," which killed trust. The fix was to either include a realistic price range with minPrice / maxPrice , or remove the offers block entirely. Partial data can be worse than no data.
Content Structure: Write for Scannable Extraction
Here's the thing about how agents read prose. They don't read it the way you do. They're looking for answers to questions, facts that can be lifted out, and clear relationships between claims.
That means your content structure should front-load the answer. If someone (or something) asks "how long does delivery take?", your page should have a heading that says something like "Delivery Times" and the first sentence under it should state the actual number. Not a paragraph of context about your warehouse operations. The number first, then the context.
I've started structuring content like this on client sites, almost like an inverted pyramid for every subsection:
- State the fact or answer directly in the first sentence under the heading
- Add one or two sentences of supporting context
- Link to a deeper resource if the topic warrants it
That's it. No warmup paragraphs. No "great question, let's explore this topic together." Agents skip that noise and sometimes misidentify where the real answer starts.
On a travel site we rebuilt last spring, we restructured 80 articles this way over about six weeks. Perplexity citations for those pages went up noticeably. More importantly, the answers those citations pointed to were actually accurate, because the relevant fact was findable.
Robots.txt, llms.txt, and Access Control
This one's newer. There's a growing convention, not yet a standard, for a file called llms.txt placed at the root of your domain. The idea, proposed by Jeremy Howard, is to give LLMs a plain-text map of your site's most important content and any access preferences. Think of it like a robots.txt but written in plain English for language model agents rather than crawl bots.
Should you implement it? Honestly, yes. It takes 20 minutes to write one. It signals that your site is agent-aware, and as more agents are trained to look for it, it'll become a meaningful signal.
On robots.txt specifically: be deliberate. Some site owners are reflexively blocking all AI crawlers now. That's your call, but blanket blocking means your content won't appear in AI-generated answers either, which is a distribution channel you're giving up. Think about it the same way you thought about blocking Googlebot in 2005. Probably not a great idea.
Internal Linking and Crawl Architecture
An agent following a workflow doesn't just land on one page. It follows links. The quality of your internal linking structure determines how much of your site an agent can actually traverse and make sense of.
Poor internal linking means agents index one or two pages of your site and stop. They don't get the full picture of what you offer.
Good internal linking means:
- Every important page is reachable within three clicks from the homepage
- Anchor text is descriptive, not "click here" or "read more"
- Related content is linked contextually in the body, not just in a widget sidebar
- Orphan pages don't exist (or if they do, you know about them and have chosen to leave them that way)
I run Screaming Frog crawls on client sites before any agent-readability work. The orphan page report alone usually reveals content that clients think is published and findable, but isn't. One client had 34 orphan pages, including their main case studies. Nobody was linking to them. Not agents, not humans.
FAQ
Do I need to restructure my entire site to be agent-readable?
No. Start with your highest-traffic pages and your conversion-critical pages. Usually that's your homepage, main service or product pages, and any content that's currently ranking and driving leads. Get those right first. A full-site audit is useful eventually but it's not where you start.
Does schema markup actually help with AI agent responses?
From what I've observed across client sites, yes. Agents that extract structured data return more accurate answers and attribute sources more reliably when schema is present. That said, it's not a magic switch. The underlying content still needs to be accurate and well-structured. Schema helps agents find and trust the data; it doesn't fix bad data.
What about sites with paywalled content?
Use the isAccessibleForFree schema property on your Article type, and the hasPart / isPartOf pattern to mark which sections are paywalled. This tells agents what they're allowed to use. Google's documentation on paywalled content structured data covers this clearly, and the same logic applies to non-Google agents.
Is llms.txt widely supported yet?
Not universally. But it costs almost nothing to implement, and early adoption of conventions like this tends to pay off. I've added it to Seahawk client sites since early 2024. It's a small signal now. It'll matter more in 12 months.
How do I test whether an agent can read my site?
Start with curl -A "Mozilla/5.0" [your URL] in a terminal and look at what comes back. If your main content isn't in that output, you have a rendering problem. Then run your pages through Google's Rich Results Test for schema validation. And check the Chrome Lighthouse accessibility audit, because agent-readability and accessibility overlap more than most people realise.
---
The web got built for humans and then retrofitted for search engines. Now it needs another retrofit, this time for agents that don't browse the way any human does. That's not a crisis. It's just the next round of work. And honestly, most of it is good practice that makes sites better for humans too. Cleaner markup, clearer content, less JavaScript sprawl. You probably should have done it anyway.