There’s a file quietly sitting at the root of some of the web’s biggest properties right now.

Stripe has one. Anthropic has one. Vercel has one. Cloudflare has one. And yes — Google has one too, across at least four of its developer properties. Which makes it all the more interesting that Google’s official documentation tells you not to bother with it.

That contradiction is worth unpacking. Because the real answer to “should you have an llms.txt?” depends entirely on a question most guides never actually answer: which AI systems use it?

This is the updated version of our original llms.txt explainer, refreshed with everything that’s changed since Google weighed in officially in May 2026.

What llms.txt actually is

llms.txt is a plain text file, written in Markdown, placed at the root of your domain at yourdomain.com/llms.txt. Its job is to give AI systems a clean, structured summary of what your site contains — without making them crawl through navigation menus, cookie banners, JavaScript payloads, and boilerplate to get to the actual content.

Think of it as a cover letter for your website, written for AI readers rather than human ones.

The format was proposed by Jeremy Howard of fast.ai in September 2024. The core idea was simple: websites are messy, and AI retrieval systems have to work hard to extract the signal from the noise. llms.txt hands them the signal directly.

A basic llms.txt looks like this:

# Crunchwiser

> SEO strategies, AI search visibility, and digital marketing analysis
  for practitioners.

## Core topics

- Local SEO
- Marketplace SEO (Amazon, Walmart, eBay)
- AI search visibility
- Technical SEO
- Schema markup

## Key content

- [Google AI optimization guide](https://crunchwiser.com/google-ai-search-optimization-guide/):
  What Google's official AI search guide means for your site
- [Marketplace SEO](https://crunchwiser.com/marketplace-optimization/):
  How product search works on Amazon and Walmart
- [Content commoditization](https://crunchwiser.com/avoid-content-commoditization/):
  Why commodity content loses in AI search

## Notes

All content reflects first-hand SEO experience and original analysis.
Written for practitioners, not beginners.

It’s human-readable, machine-parseable, and takes about an hour to write well.

The Google situation (the part everyone’s missing)

In May 2026, Google published its first official guide on optimizing for generative AI features in Search — AI Overviews, AI Mode, and everything in between. One of the things Google explicitly addressed was llms.txt.

Their position: you don’t need it.

From the documentation directly: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in generative AI search.”

That’s clear enough. But then there’s this.

While Google was publishing that guidance, Google’s own developer properties were running llms.txt files:

• ai.google.dev/gemini-api/docs/llms.txt — Gemini API docs
• developer.chrome.com/docs/llms.txt — Chrome developer docs
• firebase.google.com/docs/llms.txt — Firebase docs
• ai.google.dev/api/llms.txt — Google AI API

So Google clearly understands what llms.txt does and has decided it’s worth having for their own content. They just don’t use it as a signal in Google Search.

The reason makes sense once you think about it: Google has a massive crawling and indexing operation that doesn’t need a text file to understand your site. They’ve built that infrastructure over two decades. ChatGPT, Claude, and Perplexity don’t have that — they rely on real-time retrieval and benefit significantly from structured hints.

For a full breakdown of what Google’s guide means for SEO beyond just the llms.txt question, we covered it in detail here.

Which AI systems actually use llms.txt?

This is the question that determines whether llms.txt belongs in your setup. Here’s where things actually stand:

ChatGPT (OpenAI): ChatGPT’s browsing capability and the custom GPT builder can read llms.txt when crawling your site. Relevant for publishers whose content gets retrieved during ChatGPT searches or cited in GPT responses.

Claude (Anthropic): Anthropic has adopted the llms.txt standard. Anthropic publishes its own llms.txt and uses it when available during retrieval tasks. If you want your content correctly understood and attributed in Claude responses, having a clean llms.txt helps.

Perplexity: PerplexityBot references llms.txt when available. This one matters particularly for SEO publishers — Perplexity is a primary destination for research-heavy queries, and its citations are visible and clickable. Getting attributed correctly here is worth something.

Cursor and developer tools: The earliest llms.txt adoption came from developer documentation sites because tools like Cursor use it to navigate codebases and API docs without crawling entire sites. If your site has any developer audience, this is relevant.

Google Search: Confirmed not used for AI Overviews or AI Mode ranking. As covered above.

Bing AI / Copilot: No official confirmation of llms.txt support at the time of writing.

The pattern: llms.txt delivers value for AI systems that do active retrieval without Google-scale infrastructure. That’s ChatGPT, Claude, and Perplexity — all of which are meaningful and growing traffic sources for content-heavy sites.

What should go in your llms.txt

The spec is flexible, but the most useful files share a consistent structure. Here’s what to include:

A title and one-line description. What your site covers and who it’s for. “SEO analysis and AI search visibility for digital marketing practitioners” is more useful than “a blog about marketing.” Be specific enough that an AI can determine relevance before going deeper.

A topic list. What subjects does your site actually cover? Keep it honest. AI systems use this to route queries before retrieving individual pages.

Curated links to your best content. Not a sitemap dump. Pick your cornerstone pieces — the ones you’d want an AI to read first — and give each a one-line description. Think of it as briefing an AI researcher on where to start.

Optional sections by content type. If your site has meaningfully different content categories (evergreen guides vs. news, for example), separate them. It helps AI systems match the right content type to a given query.

A notes section. Anything an AI should know before interpreting your content. Are your articles based on original data? First-hand client experience? A specific industry perspective? Signal it here.

What to leave out: Dynamic pages, user-generated content you don’t control, pages behind logins, thin content, and anything you wouldn’t want an AI system citing as representative of your site.

llms.txt vs llms-full.txt

Some sites publish a companion file alongside the standard one: llms-full.txt.

The difference is scope. llms.txt is a curated index — it points AI systems to your content with short descriptions. llms-full.txt contains the actual full text of your content, formatted for AI consumption, so a system can read the complete material rather than just retrieve individual pages.

For most sites, llms.txt is enough. llms-full.txt makes more sense for documentation sites, research publications, or any property where an AI might need the full text rather than a pointer to a URL.

If you’re running a content publication, start with llms.txt. You can always add the full-text variant later if there’s a reason to.

How to create and publish yours

No plugin required. No complicated setup. It’s a text file.

Step 1: Open a plain text editor and write your file in Markdown. Start with a # heading (your site name), then a > blockquote (your one-line description), then your sections.

Step 2: Save the file as llms.txt.

Step 3: Upload it to your domain root so it’s accessible at yourdomain.com/llms.txt. For WordPress, this means uploading via FTP/SFTP to the same directory as your wp-config.php.

Step 4: Verify it’s live by visiting the URL directly in a browser.

A few WordPress-specific notes: some hosts block access to root-level .txt files by default through their firewall rules. If your file returns a 403, you may need to whitelist it explicitly. Also check that your robots.txt isn’t accidentally blocking your root directory — AI crawlers look for llms.txt automatically, but they won’t find it if it’s blocked.

There are also a handful of WordPress plugins emerging that auto-generate llms.txt from your published posts. Useful if you publish frequently and want the file to stay current without manual maintenance.

Does llms.txt need to be in robots.txt?

No. AI crawlers that support the standard know to look for it automatically — you don’t need to reference it in robots.txt.

What you do need to make sure: robots.txt isn’t accidentally disallowing access to your root directory or to /llms.txt specifically. Check it if you’re not sure.

Some site owners add an optional comment line in robots.txt as a signal:

# AI systems: see /llms.txt for content index

This has no technical effect on how crawlers behave. It’s just a note.

Is it worth the time?

Yes, and the time investment is genuinely low.

A well-written llms.txt takes an hour to put together. It doesn’t need frequent maintenance unless your site’s content structure changes significantly. And as AI-driven retrieval becomes a bigger share of how people find content, the number of systems that benefit from it will keep growing.

The argument against it is “Google doesn’t use it.” Which is true. But Google is no longer the only AI system sending meaningful traffic to content sites. Perplexity, ChatGPT, and Claude are real referral sources now, and they’re growing. Optimizing only for Google while ignoring the broader AI ecosystem is the same mistake people made ignoring mobile in 2012.

The other counter-argument is “AI systems crawl your site anyway.” Also true. But there’s a real difference between an AI that has to interpret your full site architecture to understand what you cover and one that gets a clean, curated brief from your llms.txt. The latter attributes your content more accurately and surfaces the right pages for the right queries.

Think of it this way: you wouldn’t skip your meta descriptions because Google can read your page anyway. llms.txt is the meta description layer for AI retrieval systems.

And given that Google itself runs llms.txt across its own developer properties, the question of whether it’s worth doing seems fairly settled.

The practical summary

llms.txt is not an SEO hack for Google Search rankings. Google was clear about that, even if their own behaviour tells a slightly different story.

What it is: a lightweight, low-effort signal that makes your content more accessible to the growing range of AI systems that retrieve and reference web content outside Google’s index. ChatGPT, Claude, and Perplexity are the ones that matter most right now.

If you’re thinking carefully about AI search visibility — and if you’re reading Crunchwiser, you probably are — llms.txt belongs in your setup. Publish it alongside your sitemap, keep it curated, and let it work quietly in the background while the AI retrieval ecosystem keeps growing around it.

Also published alongside this update: Google’s official AI search optimization guide, broken down for SEO practitioners