llms.txt Explained: What It Actually Does for Australian SEO (2026)

If someone has told you your website needs an llms.txt file or you’ll start disappearing from ChatGPT and Google’s AI Overviews, you’ve been sold half a story. The file is real. The threat isn’t. In June 2026 Google updated its own AI optimisation documentation with a section titled, more or less, “clarifying guidance on llms.txt files” — and the short version is that Google Search doesn’t use it at all, for rankings or for what shows up in AI Overviews. Gary Illyes had already said as much from the stage at Search Central Live back in July 2025. Google doesn’t support it. There are no plans to.

None of that has stopped it from becoming one of the most recommended “quick wins” in Australian SEO proposals over the past year. It costs almost nothing to add, it sounds technical and futureproof, and it gives an agency something concrete to point at in a report. We’ve had more than one client forward us a competitor’s audit with “add llms.txt” sitting near the top of the action list, phrased like a missing piece of technical SEO rather than what it actually is: an experimental file with no confirmed reader among the major search and AI platforms.

We work on AI SEO for Australian businesses every week, so we’ve followed this one closely since it first did the rounds in late 2024. This guide covers what llms.txt actually is, what’s been confirmed and what’s still just a vendor claim, where it might genuinely earn a spot on a site, and — more usefully — what to spend that same hour on if getting cited by AI tools is the actual goal.

Why This Keeps Happening: SEO’s “Add One File” Cycle

If this pattern feels familiar, it should. SEO has a long history of a single file or tag getting treated as a silver bullet long after the evidence stops supporting it. The meta keywords tag was the original version of this — webmasters in the late 1990s and early 2000s stuffed it with every term they wanted to rank for, convinced it was doing real work, years after Google had quietly stopped using it for ranking at all. More recently, rel=”canonical” and hreflang tags went through genuine hype phases where agencies oversold their impact before the industry settled into a more accurate, narrower understanding of what they actually do.

llms.txt is running the same playbook, just compressed into a shorter timeframe because the AI search narrative moves faster than the old SEO news cycle did. One commentator has already called it “the meta keywords tag of 2026”, and the comparison holds up: low effort to implement, intuitively appealing, heavily promoted by tool vendors with a commercial interest in the trend, and — so far — unsupported by the platforms whose support would actually matter. None of that makes the underlying idea stupid. Jeremy Howard’s original pitch, giving AI systems a cleaner way to parse large documentation sites, is reasonable on its own terms. It just hasn’t translated into the broad SEO or AI-citation benefit it’s being sold as.

What llms.txt Actually Is

llms.txt is a plain Markdown file published at a predictable location on a website — yoursite.com.au/llms.txt — containing a short, curated list of a site’s most important pages, usually with a one-line description of each. It was proposed on 3 September 2024 by Jeremy Howard, co-founder of the AI research lab Answer.AI, as a lightweight way to hand a large language model a map of a site’s content instead of making it crawl and parse full HTML pages to figure out what matters.

The idea made intuitive sense at the time. Large language models work far more efficiently with clean Markdown than with HTML cluttered by navigation menus, cookie banners, embedded scripts and ad markup. A documentation site with hundreds of pages could, in theory, hand a model a tidy index instead of forcing it to wade through templated chrome on every page. Howard’s own projects adopted it quickly, and a wave of developer tools, static site generators and documentation platforms followed.

How It Differs From robots.txt and sitemap.xml

This is where most of the confusion starts, because llms.txt sits in the same folder as two files that genuinely do matter, and a lot of people assume it works the same way.

robots.txt is a crawling instruction. It has been part of the web since 1994, every major search engine and well-behaved crawler respects it, and it tells bots which parts of a site they’re allowed to request. It’s not a security control — a crawler that ignores it simply ignores it — but it is a long-standing, universally followed standard.

sitemap.xml is a discovery aid. It lists the URLs on a site that are worth crawling and indexing, along with metadata like last-modified dates. Google and other search engines treat it as a strong hint rather than a binding rule, but in practice it’s one of the most reliable ways to get new or updated pages found quickly, particularly on large sites.

llms.txt does neither job. It doesn’t restrict access like robots.txt, and unlike sitemap.xml, there’s no confirmed major platform that systematically fetches and uses it to decide what to crawl or cite. It’s a proposal, not a standard, and as of 2026 it still hasn’t been adopted by the organisations whose adoption would actually make it matter.

What a typical llms.txt file looks like

In practice it’s a single plain-text file, written in Markdown, that opens with an H1 for the site or product name, a one-line summary underneath, and then a series of headed sections — often something like “Docs,” “Guides” or “API Reference” — each containing a short bulleted list of links with a brief description beside each one. There’s no required schema, no validator that enforces a correct structure, and no character limit. That looseness is part of the appeal — anyone can write one in ten minutes — but it also means there’s no consistency across sites for a model to learn to rely on, which is part of why adoption by AI platforms has stayed thin.

What Google (and Everyone Else) Actually Says in 2026

We’d rather point to what’s actually been confirmed than repeat marketing copy, so here’s where things stood through 2026.

Google’s position, restated and formalised in its June 2026 AI optimisation guidance update, is that Search doesn’t use llms.txt for anything — not crawling priority, not ranking, not AI Overviews inclusion. This wasn’t a new policy; it was Google putting in writing what Gary Illyes had already said publicly nearly a year earlier, when he confirmed at Search Central Live that Google has no plans to support the file.

Independent research backs this up. Ahrefs ran a study across more than 137,000 domains and found that among sites with a valid llms.txt file, 97% received zero requests for that file during the study period in May 2026. Put plainly: almost no one, human or bot, was even fetching the file to begin with, let alone acting on its contents. Separately, SE Ranking looked for a correlation between having an llms.txt file and appearing more often in AI-generated citations, and found none worth reporting.

Where the picture gets murkier is around the AI chat platforms themselves rather than Google. Some vendor blogs claim Perplexity and Claude will read an llms.txt file if a user’s query happens to trigger a live fetch of that domain. That claim is plausible in isolated cases — both companies have published documentation discussing the format with some interest — but it’s a long way from “adding this file gets you cited more often.” No AI platform has published data showing llms.txt adoption changes citation frequency, and most of the sources making confident claims either sell AI-visibility tracking tools or GEO (generative engine optimisation) retainers, which is worth factoring in before taking their enthusiasm at face value.

If your goal is showing up in Google Search or Google’s AI Overviews specifically, the evidence through 2026 says skip it. That advice might shift if a major platform announces genuine, confirmed support — but that hasn’t happened yet, and “might one day” isn’t a reason to prioritise it over work with a known payoff.

Where llms.txt Might Still Earn Its Place

None of this means the file is worthless everywhere. It just means the case for it is much narrower than the pitch most businesses hear.

Developer documentation and API references

If a business runs a documentation site, an API reference, or a developer-facing knowledge base, there’s a genuine (if still unproven at scale) argument for llms.txt. Developers increasingly paste documentation URLs into AI coding assistants and ask them to work from the content directly. A clean Markdown index can make that easier for tools that do choose to fetch it, and the file costs almost nothing to maintain if the underlying docs are already well-organised. This is also the context Jeremy Howard built the format for in the first place — it was never really pitched as a marketing SEO tactic. An Australian SaaS company with a public API, for instance, has a genuinely different calculus here than a local plumber’s website, because the audience reading its docs increasingly includes AI coding tools acting on a developer’s behalf, not just humans browsing in a tab.

Large, well-structured content sites with the resources to maintain it

A content-heavy site with a genuinely large page count — think a publisher, a knowledge base, or a large multi-category blog — might find some value in a curated index, mostly as a hedge in case platform support materialises later. The operative word is “curated.” A good llms.txt file needs real editorial judgement about which pages matter, and it needs updating every time that priority list changes. A stale llms.txt pointing to deleted pages or missing a site’s newest, most important content is arguably worse than having none, since it actively misdirects whatever does happen to read it. In practice this usually means a marketing team, a content lead, or an agency with ongoing visibility into the site’s structure, rather than a one-off task ticked off during a build and forgotten about.

Where it almost certainly isn’t worth the time

For the overwhelming majority of Australian small and medium businesses — service businesses, ecommerce stores, local clinics, trades, professional services — llms.txt sits well down the priority list. These sites don’t have hundreds of documentation pages that benefit from a curated Markdown map; they have a home page, a handful of service pages, maybe a blog, and the real work is making those pages genuinely clear, well-structured and backed by legitimate authority signals. Spending budget on llms.txt instead of that work is, at best, a wash, and at worst a distraction dressed up as progress.

An Australian ecommerce store is a good example of where the mismatch shows up clearly. A product catalogue with thousands of SKUs isn’t the kind of “curated, high-value page list” llms.txt was designed around — you can’t sensibly hand-pick a few dozen product pages out of a few thousand and call it representative, and generating the file automatically just reproduces the sitemap. The far more useful investment for a store chasing AI visibility is structured product data (price, availability, reviews) via schema, and category pages that read as genuinely useful buying guides rather than thin lists. Neither of those needs an llms.txt file to work.

What Actually Moves the Needle for AI Citation in Australia

We’ve written a broader guide to getting cited by ChatGPT, Google AI Overviews and Perplexity that covers the full strategy end to end. This section is narrower on purpose: it’s the short version of what to prioritise instead of llms.txt, specifically, if a competitor’s audit has put the file at the top of your list.

Structured data that actually describes your business

Schema markup — Organization, Service, FAQPage, LocalBusiness and the rest — gives both traditional search engines and AI systems an unambiguous, machine-readable description of what a page is about, who it’s for, and how the pieces relate to each other. Unlike llms.txt, schema has years of confirmed use by Google for rich results, and growing (if less precisely measured) use by AI systems trying to extract clean facts from a page. We go into the detail of which schema types actually earn visibility in our schema markup guide, but the short version is: this is real, confirmed, structured machine-readability work, and it’s the closest thing to “llms.txt done properly” that actually has evidence behind it.

Clear page structure and genuinely useful FAQ content

AI systems that answer questions directly — rather than just linking out — tend to lift content that’s already structured as a clear answer to a specific question. A page with a proper heading hierarchy, a direct answer near the top of each section, and an honest FAQ block (not a padded one stuffed with keyword variations) gives a model something easy to extract and attribute. This is standard on-page structure work, the kind that’s been good SEO practice for a decade, and it happens to be exactly what AI citation research keeps pointing back to.

Topical authority and real-world citations

Both Google’s AI Overviews and the major chat assistants lean heavily on sources that already show up independently across the web — industry directories, press coverage, other sites linking to yours, genuine third-party mentions. This is the part no file on your own server can manufacture. It’s slower and harder than adding a text file, which is exactly why it’s worth more: it’s not something a competitor can copy by reading the same blog post you did.

A site that’s actually fast and crawlable

It’s easy to overlook the basics once the conversation turns to AI, but a site a model (or a traditional crawler) can’t fetch quickly and cleanly is a site that loses out regardless of how well-written an llms.txt file sitting next to it might be. Server response times, a sane URL structure, and pages that render their core content without depending on heavy client-side JavaScript all affect whether a crawler bothers coming back often, and whether an AI system fetching a page live gets something usable within its time budget. This is unglamorous work compared to a trending AI SEO tactic, but it’s foundational to all of it.

Should You Add llms.txt Anyway? A Practical Decision Framework

Given it costs very little to add, the honest answer for most businesses is: it won’t hurt, but it also won’t help, and it shouldn’t come before anything with a confirmed payoff. Here’s how we’d actually decide, in order.

First, ask whether the site is documentation-heavy — API references, developer guides, long-form technical content that’s updated often and genuinely benefits from a curated index. If yes, it’s a reasonable low-cost addition, provided someone actually owns keeping it current.

If the site isn’t documentation-heavy, the next question is whether the fundamentals are already sorted: titles and meta descriptions written properly, schema markup in place, Core Web Vitals passing, content that actually answers the questions people are searching. If those basics still need work, that’s where the time and budget should go, full stop. llms.txt has no confirmed effect on rankings or AI citation; spending an afternoon on it ahead of a technical SEO fix is choosing the activity that feels productive over the one that is.

If the fundamentals genuinely are already handled and there’s a spare hour going spare, adding the file does no harm. Just don’t expect to see it move anything, and don’t let anyone bill it as a meaningful AI SEO deliverable on its own.

If you do add one, build it properly

A usable llms.txt file is short, plain Markdown, and genuinely curated rather than a dump of every URL on the site. Include a one-line description of the business, then a short list of the pages that actually matter — key service pages, your most substantial guides, anything you’d want a model to understand first if it only read one page. Skip boilerplate pages like your privacy policy or contact form. The format is deliberately simple; there’s no benefit to making it elaborate.

A reasonable structure for a service business looks something like an H1 with the business name, a one-line summary sentence underneath, then an “## Services” heading with three to six links to the actual service pages, and optionally a “## Guides” heading linking to the handful of blog posts that genuinely represent the business’s expertise — not all of them, just the strongest few. Resist the urge to list every page the site has. The entire value of the format, to the extent it has any, comes from the curation. A list of everything is just a worse sitemap.xml.

The stale-file risk nobody mentions

The failure mode that gets skipped in most of the enthusiastic write-ups is what happens after launch. Sites get redesigned, pages get merged or retired, service offerings change, and an llms.txt file written once during a 2026 site build quietly becomes wrong. A stale file pointing a model toward a page that no longer exists, or missing the service you actually want surfaced, is arguably worse than skipping the file altogether — it’s confidently wrong rather than simply absent. If you’re not committing to reviewing it every time the site structure changes, you’re better off not adding it in the first place. This is exactly the kind of small, easy-to-forget maintenance item that falls through the cracks without an ongoing website care and support arrangement — not because it’s hard, but because nobody remembers it exists until something’s already gone stale.

Common Mistakes We See With llms.txt

Most of the problems we see aren’t with the file itself — it’s genuinely low-risk — but with the decisions made around it.

• Treating it as a ranking factor or an AI Overviews inclusion lever, when Google has explicitly confirmed it isn’t one.
• Adding it once during a site build and never touching it again, so it drifts out of sync with the real site within a few months.
• Confusing it with robots.txt and assuming it can restrict what AI tools access — it has no enforcement mechanism at all.
• Paying an agency for an “AI SEO” deliverable that turns out to be little more than this one file, with no schema, no content structure work and no citation strategy behind it.
• Prioritising it over genuine technical SEO or content fixes because it’s quicker to tick off and easier to put in a report.
• Assuming that because a vendor blog says Perplexity or Claude “can” read the file, they reliably do, for your specific site, in a way that changes outcomes.
• Generating the file automatically from every URL on the site, rather than curating it, which produces a bloated list that defeats the entire point of a short, prioritised index.
• Reporting its presence as a KPI or deliverable milestone, when there’s no metric on the other side — no ranking movement, no citation count — that it can be tied back to.

Frequently Asked Questions

Where to Start

If an audit or a proposal has put llms.txt near the top of your to-do list, that’s worth a second look before you spend time on it. It’s not going to hurt anything, but it’s also not where the actual gains in AI visibility or search rankings are sitting in 2026. The honest priority order is: get your technical SEO and schema markup genuinely right, structure your content so it answers real questions clearly, build the kind of external authority AI systems and search engines both lean on, and treat llms.txt as an optional, low-effort extra for documentation-heavy sites rather than a strategy on its own.

This is also, honestly, a decent test of any agency or consultant you’re evaluating for AI SEO work. Ask them specifically what llms.txt is supposed to do for your business, and listen for whether the answer is grounded in anything Google, Perplexity or OpenAI has actually confirmed, or whether it’s just a line item that sounds current. The same scrutiny is worth applying to every AI SEO tactic right now — the field is moving fast enough that genuine research and marketing hype are easy to mix up, even for people who aren’t trying to mislead anyone.

If you want an honest read on where your own site actually stands — what’s genuinely moving the needle versus what’s just padding a report — our AI SEO packages start with a real audit of both, at a fixed monthly price with no lock-in contract.

Worth putting a number on it: our AI SEO packages start from $800 a month, including the audit, keyword research, on-page and technical work, schema and ongoing tracking across both search rankings and AI visibility. That’s below the $2,000-plus a month that full-service AI SEO and generative engine optimisation retainers commonly run to in Australia for comparable scope, once you’re past basic DIY tracking tools. If you’ve already got a written quote from another agency for comparable scope, send it through and we’ll do our best to beat it.

Either way, happy to just talk it through first. Tell us what’s actually in your current SEO or AI SEO scope and we’ll give you a straight answer on what’s worth keeping, what’s worth dropping, and whether llms.txt deserves a place in it at all.