In 1994 robots.txt let publishers tell AltaVista and, later, Google what to skip. Three decades on, large-language-model (LLM) crawlers need different guidance: not every URL is helpful during training, and some material is premium yet still valuable at answer time. A draft standard called llms.txt fills that gap. Search Engine Land sums it up as “robots.txt for AI”, noting its steady rise since March 2025. With holiday code freezes only a few months away, now is the right moment for Australian site owners to prepare their file, test it, and avoid a spring scramble.
Why robots.txt falls short
Robots.txt speaks in simple allow/deny rules aimed at deterministic search spiders. LLM bots, by contrast, juggle two activities: creating training corpora and fetching live context for retrieval-augmented answers. The original standard can block them entirely but cannot say, for example, “Please quote these tutorials but don’t store pay-walled reviews.” The llms.txt proposal, drafted by Jeremy Howard and open for comment on llmstxt.org, adds exactly that nuance, plus optional pacing instructions and content priorities.
What llms.txt actually does
Placed at https://yourdomain.com/llms.txt, the file is plain Markdown that a human can skim and a model can parse without tripping over navigation bars or ad scripts. Popular documentation host Mintlify calls it “a lightweight summary of your most important content” and auto-generates the file for every site it hosts. Unlike a sitemap, which lists everything, llms.txt is curated; unlike robots.txt, it can flag pages for training, inference, or neither.
Anatomy of a minimal file
Key directives appear below:
| Directive | Purpose | Example |
| Allow / Disallow | Permit or block a path | Disallow: /drafts/ |
| Training | allow or disallow model training | Training: allow |
| Inference | Control live answering access | Inference: allow |
| Crawl-Delay | Seconds between requests | Crawl-Delay: 10 |
| Priority | 0–1 weight for answer selection | Priority: 0.9 /docs/faq.md |
Early tests at Perplexity show that pages marked High-Quality with a priority at or above 0.8 surface more often in answer footnotes than comparable, untagged content.
Six practical steps to finish by October
- Audit your site. Mark confidential areas and pages you actively want quoted.
- Add Markdown twins. For reference articles, post .md versions to cut HTML noise, as the llmstxt.org spec recommends.
- Draft the file. Keep comments concise; use UTF-8 plain text.
- Validate. A lint script in the llms.txt GitHub repo catches misplaced directives and missing colons.
- Deploy at the root. Publish and purge CDN caches.
- Monitor access logs. Note who respects or ignores rules and adjust User-agent lines as new crawlers appear.
Who is already shipping it?
- FastHTML project – Its llms.txt links to quick-start guides and auto-generated .md docs so code assistants can cite the library precisely.
- Australian newsrooms – Two major mastheads mark pay-wall articles as training-disallowed while leaving summaries open for inference, protecting subscription value.
- Tech directories – A community list by llmstxt tracks more than 70 live files and grows weekly.
Tool support is arriving fast
WordPress site owners need not write the file by hand. At least three plugins—Website LLMs.txt, LLMs.txt Generator, and Yoast’s new AI module—can build and refresh it automatically. Static-site engines Hugo and Docusaurus already have generator scripts, and Mintlify bundles llms-full.txt for anyone who wants a single mega-context file ready to drop into Claude or ChatGPT.
Legal and commercial upsides
Australian businesses currently rely on a patchwork of copyright law and contract terms when challenging unlicensed AI usage. Publishing an explicit opt-in or opt-out signal now may strengthen any future claim and demonstrates reasonable steps to protect content. It also smooths commercial talks: showing potential licensing partners a neat llms.txt file is easier than emailing spreadsheets of URLs.
All in all, an llms.txt file is short, but the control it delivers when combined with generative engine optimisation can be long-lasting. Drafting one ahead of Q4 avoids late-season risk, positions your content for clearer citation in chat answers, and signals to regulators that you take AI governance seriously. Set aside half a day this winter: run the audit, generate clean Markdown where it matters, add the directives, and ship. By the time Black Friday kicks off, both humans and language models will see your site exactly the way you intend.
Frequently Asked Questions
- What is llms.txt and how is it different from robots.txt?
llms.txt is a plain-text policy file that instructs large-language-model crawlers on training, inference, crawl pacing and content priority. Robots.txt only allows or disallows search spiders. llms.txt adds directives such as Training, Inference and Priority, letting publishers permit quotations yet block dataset storage, all while throttling heavy bots during peak events. - Where should I place the file and what format does it require?
Host llms.txt in your site’s root directory—https://yourdomain.com/llms.txt—so crawlers find it automatically. Save the document in UTF-8 plain text or Markdown, avoiding HTML headers. Use Unix line endings, single‐byte spacing, and comment lines beginning with #. Upload via your normal deployment pipeline, then purge CDN caches immediately after each major update. - How often should llms.txt be updated, and what monitoring is advisable?
Review llms.txt quarterly or whenever you launch a new content section, change pay-wall rules, or notice unfamiliar bots. Keep server logs and web-server analytics open; flag user-agent patterns ignoring directives. Updating promptly preserves legal posture, reduces bandwidth spikes, and ensures priority scores reflect your most current authoritative resources at scale.
