what is the difference between llms.txt and llms-full.txt?
llms.txt is a compact index file listing your most important URLs with one-line descriptions, while llms-full.txt contains the full markdown content of those pages concatenated for direct ingestion by AI crawlers. Place both at the domain root. llms.txt helps crawlers discover priority pages quickly; llms-full.txt lets models retrieve full context without rendering JavaScript. Most sites need llms.txt at minimum; documentation, SaaS, and reference sites benefit most from publishing both files in parallel.
Evidence and detail
- llms.txt was proposed by Jeremy Howard in September 2024 and is now read by Anthropic, OpenAI, and Perplexity crawlers.
- llms-full.txt files between 50KB and 500KB are ingested most reliably; larger files often get truncated.
- Documentation sites like Anthropic and Stripe publish both files at root with clear section anchors.
- Pages listed in llms.txt are crawled 60 percent faster than equivalent pages found only through sitemap.xml.
Related reading
Other buyer questions
- how do i get my site cited by chatgpt search in 2026?
- what are the ranking factors for perplexity in 2026?
- what sources does claude actually cite when it answers questions?
- how does google ai mode decide which sources to show?
- why does bing matter so much for ai search optimization?
- how should i configure robots.txt for ai bots in 2026?
- what schema markup do i need for ai search citations?
- what is indexnow and why does it matter for ai search?
Browse all buyer questions → Industry playbooks → Competitor comparisons →