llms.txt is a markdown file at the root of a domain that provides a structured, machine-readable summary of a site's content for AI crawlers. Unlike robots.txt (which controls access) and sitemap.xml (which aims for completeness), llms.txt prioritizes which pages matter most for answering questions about the brand.

Should I block AI crawlers in robots.txt?

Blocking all AI crawlers prevents RAG retrieval and real-time citations. The optimal configuration for brands seeking AI citations is the crawl-no-training approach: allow AI crawlers like GPTBot, ClaudeBot, and GeminiBot for retrieval access, while protecting training rights through legal and technical mechanisms outside of robots.txt.

Which AI crawlers should I allow?

The most important crawlers to allow are GPTBot (ChatGPT/Bing), ClaudeBot (Claude/Brave), GeminiBot (Gemini/Google), CCBot (Common Crawl), and PerplexityBot (Perplexity). Each enables real-time RAG retrieval on a major AI platform.

How does llms.txt improve entity resolution?

Models that support the llms.txt protocol treat the file's content as an explicit entity declaration. Combined with Organization schema and Wikidata, llms.txt forms the third link in an entity declaration chain that tells models what your brand is, what content matters, and how to prioritize it.

The llms.txt & robots.txt Playbook for AI Crawlers

Key Takeaways

llms.txt is the highest-impact, lowest-effort GEO change — it takes 2 hours to create and immediately gives AI crawlers a structured map of your site.
Most enterprise sites block more AI crawlers than they realize — default configurations block GPTBot, ClaudeBot, and GeminiBot, preventing RAG retrieval.
The optimal configuration is crawl ⇢ no-training — allow retrieval while protecting training rights through legal mechanisms.
llms.txt and robots.txt serve complementary functions — one controls access, the other controls prioritization. Most sites configure only one.
llms.txt is becoming an entity resolution signal — models treat it similarly to Organization schema for entity classification.

Two files — llms.txt and robots.txt — determine whether your content is accessible to AI crawlers for RAG retrieval, and whether it is prioritized in the retrieval set. Most enterprise sites have one well-optimized and the other ignored. Few have both configured for AI citation goals.

What llms.txt Is (and Why It Exists)

The llms.txt specification defines a markdown file at the root of a domain that provides a structured, machine-readable summary of a site's content for AI crawlers. Think of it as a sitemap specifically designed for LLMs — not for listing every page, but for prioritizing which pages matter most for answering questions about the brand.

A well-structured llms.txt includes: a brief description of the organization, links to core pages organized by category, optional per-page descriptions telling the crawler what each page covers, and direct links to content that should be prioritized for retrieval.

Unlike robots.txt, which is about access control, and sitemap.xml, which is about indexing completeness, llms.txt is about content prioritization. It tells the crawler: "These are the pages that best represent what our brand is and what we do."

The Status Quo Problem

Most enterprise sites have a well-optimized robots.txt file, a properly structured sitemap.xml, and no llms.txt at all. This configuration was designed for the search engine era and is actively counterproductive for AI citation.

Without llms.txt, AI crawlers must discover and prioritize your content through general-purpose crawling — meaning your most citable pages compete equally with your least citable pages, and the crawler has no machine-readable signal about what your brand is or which content represents it best.

How to Structure Your llms.txt

The structure follows a standard format: a level-1 heading with your brand name, a one-paragraph entity description, then section headings with bulleted links to key pages. Use the exact same entity description as your Organization schema's description field and your Wikidata entry. Consistency across these three sources creates the strongest possible entity anchor.

List no more than 20-25 links. The file is about prioritization, not completeness. Order links by citation value, not by site structure. Include at least one comparison or product page if available. Update the file when you publish content designed for AI citation.

robots.txt for AI Crawlers

The robots.txt file controls which crawlers can access which parts of your site. For AI citation, the default configuration creates a tension: blocking AI crawlers protects your content from training but also blocks it from RAG retrieval.

The Crawl ⇢ No-Training Configuration

The optimal configuration allows AI crawlers to access content for real-time RAG retrieval (necessary for citation) while protecting training rights through legal mechanisms:

User-agent: GPTBot
Allow: /

Apply the same Allow pattern for ClaudeBot, GeminiBot, CCBot, and PerplexityBot. Each enables retrieval on a major AI platform.

The Blocking Tradeoff

Blocking all AI crawlers protects content from unauthorized training but has three costs: content is unavailable for RAG retrieval (preventing real-time citations), content is excluded from Common Crawl (reducing training data prevalence), and models cannot verify entity information against your domain (weakening entity resolution). The recommended approach for brands seeking AI citations is the crawl-no-training configuration: allow retrieval, protect training rights through other mechanisms.

How the Two Files Work Together

robots.txt controls access — a binary gate. llms.txt controls prioritization — a selective guide. Without robots.txt, crawlers may not reach your content. Without llms.txt, crawlers that do reach your content may not find your best citations.

Together with Organization schema, these files form an entity declaration chain: Organization schema tells the model "this domain is entity X," llms.txt adds "entity X covers these topics and prioritizes this content," and the Wikidata entry contributes "entity X has these properties and relationships."

Measuring Impact

After implementing both files, track: whether AI crawlers are hitting your llms.txt file (check server logs for requests from GPTBot, ClaudeBot, etc.), whether citations of prioritized pages increase relative to non-prioritized pages, and whether entity classification accuracy improves.

In our analysis, brands that added a well-structured llms.txt file saw a median 40% increase in citations of pages listed in the file within 60 days, compared to non-listed pages on the same sites.

Check Your AI Crawler Configuration

GetCiteFlow's scanner checks whether your robots.txt allows the right crawlers, whether your llms.txt is properly structured, and whether your entity declaration chain is consistent across all three layers. Free scan.

Get Your Free AI Visibility Scan