Both files live at the root of your domain, both are plain text, and both have something to do with bots. That is where the similarity ends. robots.txt is an access-control file that tells crawlers which URLs they may fetch. llms.txt is a content file that hands AI models a clean, curated map of what matters on your site. One says stay out; the other says start here.
Key takeaway: robots.txt controls what crawlers are allowed to fetch, while llms.txt curates which content you want AI models to read first. They do not overlap, so most sites should ship both.
What robots.txt actually does
robots.txt has been around since 1994 and is now a formal standard, RFC 9309. It is a set of allow and disallow rules grouped by user-agent. When a well-behaved crawler arrives, it reads robots.txt first and skips anything you have disallowed. It is a crawl directive, not a security boundary: it asks bots not to fetch a path, it does not stop a determined one, and it does not by itself remove a page from an index.
The practical uses are narrow and well understood: keep crawlers out of faceted URL parameters, admin paths, and API routes, and point them at your sitemap. If you want a page out of Google, you use a noindex tag or a removal, not a robots disallow, because a disallowed page can still be indexed from external links.
What llms.txt actually does
llms.txt is far newer. It was proposed in September 2024 as a Markdown file at /llms.txt that gives large language models a concise, link-rich index of your most useful pages. Think of it as a hand-built table of contents for your site, written for inference time rather than crawl time. Instead of a model guessing which of your 2,000 URLs explain your product, you list the canonical ones in priority order, with short descriptions.
The honest position in 2026: llms.txt is a proposal with real momentum and growing tool support, but the major AI providers have not all committed to reading it, and there is no equivalent of RFC 9309 behind it yet. I treat it as cheap, low-risk upside. It costs an afternoon, it cannot hurt your SEO, and it puts your best content in front of any model that does choose to use it. For the full how-to, see my [llms.txt explainer](/blog/llms-txt-explained-2026/).
The differences that matter
Job: robots.txt restricts access; llms.txt recommends content. Format: robots.txt uses its own allow/disallow grammar; llms.txt is plain Markdown with headings and links. Timing: robots.txt is read at crawl time by search bots; llms.txt is meant for retrieval and inference by language models. Enforcement: robots.txt is widely respected by search engines; llms.txt is advisory and adoption is still uneven. Risk of getting it wrong: a bad robots.txt rule can deindex your whole site; a bad llms.txt does nothing worse than get ignored.
Do they conflict? Should you have both?
They do not conflict, because they operate on different layers. robots.txt can still block an AI crawler at the fetch level (Google-Extended, GPTBot, and others are user-agents you can disallow), while llms.txt curates content for the models that do read your site. If you block a crawler in robots.txt, that decision wins regardless of what llms.txt says, because the bot never gets far enough to read the curation.
For most content sites the answer is simple: keep a tight robots.txt that protects parameters and admin paths and exposes your sitemap, and add an llms.txt that lists your genuinely important pages. If your strategy is to keep AI models out entirely, that is a robots.txt and user-agent decision, not an llms.txt one.
FAQ
Does llms.txt replace robots.txt?
No. They do different jobs. robots.txt controls which URLs crawlers may fetch; llms.txt suggests which content AI models should prioritise. Removing robots.txt to add llms.txt would strip your crawl controls and leave parameters and admin paths exposed.
Can I block AI crawlers with llms.txt?
No. Blocking is a robots.txt job. To keep AI crawlers out, disallow their user-agents (such as GPTBot or Google-Extended) in robots.txt. llms.txt has no access-control function at all; it only curates content for models that already read your site.
Where do both files go?
Both sit at the root of your domain: /robots.txt and /llms.txt. They are served as plain text and Markdown respectively, and you can ship and update them independently.
Will llms.txt help my SEO?
Not directly. It is aimed at AI answer engines, not Google ranking. The realistic upside is generative-search visibility: if a model uses your llms.txt, it finds your best pages faster. It will not move classic blue-link rankings, and it cannot hurt them.