llms.txt: Should You Create One for Your B2B Site?

TL;DR

• For a B2B marketing site, an llms.txt file is not a priority. No measurement we read shows it increases AI citations, and Google says Search ignores it.
• Three things move your visibility in AI answers far more: AI crawler access in robots.txt, pages that render for bots, and clear pages about your company.
• If you want one anyway: an hour of work, a minimal template, no sensitive data, a quarterly review, a check in your server logs.
• The exception: technical documentation or an API.

According to DataForSEO (September 2026), "llms txt" is searched about 5,400 times a month in the US (Google Ads, range of 3,600 to 5,400 over 12 months, 4,400 in September 2026). Between citation promises and "useless" verdicts, here is a decision framework for executives, with its limits.

What is an llms.txt file?

An llms.txt file is a Markdown file served from the root of a website (for example yoursite.com/llms.txt) that summarizes the site and lists its key pages for an AI system. Jeremy Howard proposed it in September 2024. The reference page, llmstxt.org, is now at version 2 (last modified August 10, 2026).

The format is light: only an H1 with the site name is required, then a blockquote summary, H2 link lists and an "Optional" section for secondary links.

It is a proposal, not a standard, and it is not access control. Per llmstxt.org, it is mainly useful at inference time (when an assistant answers a question), not for training, and it complements robots.txt rather than replacing it.

• robots.txt: What it does: Access rules: which crawlers may fetch which pages; Who honors it: Google, OpenAI, Anthropic and Perplexity say they respect it (with exceptions for user-⁠triggered fetches at some of them)
• sitemap.xml: What it does: Lists the URLs you want crawled; Who honors it: Established standard that Ahrefs still ranks above llms.txt in importance
• llms.txt: What it does: Voluntary Markdown map of your key pages; Who honors it: No major AI company has confirmed using it for search (Ahrefs, Semrush); Google Search ignores it

Is llms.txt actually used? What the server logs show

Who publishes it: mostly technical documentation

Microsoft's Teams SDK docs publish llms.txt files "optimized for AI coding assistants" (page updated July 28, 2026). Documentation platforms such as Mintlify generate them. On adoption, 28% of 137,210 domains in Ahrefs' panel publish one as of May 2026. The panel skews technical, so Ahrefs treats that as a likely upper bound.

Who reads it: almost no one, according to logs

Publishing a file does not prove anyone reads it. Here are four log measurements:

• Ahrefs, June 2026: Sample: 137,210 domains, May 2026; Result: 97% of valid files received zero requests; the rest drew about 22,000 requests, 19.5% of them from AI bots; Limit: A request is not a read; panel skews technical
• SeoMix, June 2026: Sample: Logs from 6 sites, 60 days (April and May 2026), 33,092 AI bot visits; Result: No request for llms.txt; Limit: 6 sites, 2 months
• Semrush, September 2026: Sample: Search Engine Land, mid-⁠August to late October 2025; Result: Zero visits from GPTBot, PerplexityBot and ClaudeBot; Googlebot and Bingbot, a few hits; Limit: One site, bot verification not described
• OtterlyAI, via Search Engine Land, October 2026: Sample: 90-day log study; Result: 84 of 62,100 AI bot requests hit /llms.txt (about 0.1%); Limit: Reported figures, original study not opened

So who does fetch it? Coding agents: Ahrefs found that Claude Code ranks among the named clients of the file, ahead of AI search bots. Agents fetch it when pointed to it (a link, docs, an instruction), and AI bots do not go looking for a missing file: none of the requests that got a 404 came from an AI bot.

What is proven about llms.txt and AI visibility, and what is not

Here is what the major players say publicly, based on the official pages we opened on October 26, 2026:

• Google Search: Its AI optimization guide (updated July 10, 2026): "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search." Google Search ignores them: no help, no harm
• Google Chrome: The Lighthouse llms.txt audit calls it "an emerging convention" and treats it as optional: a 404 is "not applicable"
• John Mueller (Google): Asked on Bluesky whether Google hosting llms.txt files on its own sites was an endorsement, he answered: "I'm tempted to say something snarky since this has come up so often, but to be direct, no." (Search Engine Roundtable, January 20, 2026)
• OpenAI, Anthropic, Perplexity: Their pages on their crawlers (OpenAI, Anthropic, Perplexity) describe each bot and how robots.txt controls it. None presents llms.txt as a signal (OpenAI and Perplexity only mention it as the index of their own docs)

None of these pages says llms.txt is read to pick cited sources. Silence is not proof, but there is nothing to buy on that basis.

What matters before llms.txt: three fixes

1. AI crawler access in robots.txt

Each company runs search, training and on-⁠demand bots that you can set separately:

• OpenAI: Search bot: OAI-⁠SearchBot: if blocked, your site is not shown in ChatGPT search answers; Training bot: GPTBot: if blocked, content is excluded from training; On-⁠demand fetch: ChatGPT-⁠User: robots.txt rules may not apply
• Anthropic: Search bot: Claude-⁠SearchBot: if blocked, reduced visibility in results; Training bot: ClaudeBot: if blocked, content is excluded from training; On-⁠demand fetch: Claude-⁠User: if blocked, reduced visibility
• Perplexity: Search bot: PerplexityBot: Perplexity recommends allowing it to appear in results; Training bot: No training use, per Perplexity; On-⁠demand fetch: Perplexity-⁠User: generally ignores robots.txt
• Google: Search bot: Googlebot; Training bot: Google-⁠Extended: robots.txt token for Gemini, no separate user agent, no effect on Search; On-⁠demand fetch: Not applicable

Settings are independent and can take about 24 hours to apply (OpenAI, Perplexity). Allowing search bots while deciding separately on training bots is a content policy choice. The foundation (indexing, canonical, sitemap, rendering) belongs to technical SEO.

2. Rendering and crawlability

A Vercel study from December 2024 found that none of the major AI crawlers of the time (OpenAI, Anthropic, Perplexity) executed JavaScript, while Gemini, which uses Googlebot's infrastructure, could. Vercel's advice: server-⁠render critical content. The study is dated: verify on your own site.

The simplest test: view the source of a service page. If your offer is not in the raw HTML, a bot that does not run JavaScript cannot see it. Google also tells publishers to keep content crawlable. Our SEO audit guide covers these checks: robots.txt, content visible without scripts, sitemap.

3. Clear pages about your company

Google's guide recommends unique content your visitors find satisfying, and notes that structured data is not required for generative AI features but remains useful for rich results. Our reading: one page per offer that says who you are, what you sell, for whom and with what proof, and the same facts everywhere (site, profiles, directories). The full method is in our guide to ranking in ChatGPT.

Can AI crawlers actually read your site? We check robots.txt, rendering, sitemap and entity pages before talking about files. Get your free action plan

How to measure llms.txt with your server logs

An AI can cite your brand without ever reading a suggestion file. Only your server logs show who visits what. Here is a five-⁠step method:

1. 01Pull 30 to 60 days of raw logs from your host, CDN or web application firewall.
2. 02Filter on the user agents the vendors publish (the ones in the table above, plus Googlebot). Google-⁠Extended does not show up in logs.
3. 03Verify identity: Anthropic and Perplexity publish their IP addresses, which lets you spot impostors.
4. 04Count three things: bot requests on your key pages (status 200), requests on /llms.txt (200 or 404) and 4xx or 5xx errors returned to AI bots.
5. 05Add two outcome measures: AI-⁠assistant referrals in your analytics tool and a fixed panel of questions you ask the assistants each month.

Our decision rule: if 90 days after launch no AI search bot has requested the file, stop spending time on it. And remember Ahrefs' caveat: a request is not a read.

A minimal llms.txt example for a B2B site

If you decide to create one, keep it short. Our recommendation, which is an opinion and not a spec: 10 to 20 links to your most durable pages, with one sentence per link saying what the page covers and who it is for. Search Engine Land gives the same advice: evergreen pages, not your whole sitemap.

# Your Company

> One clear sentence: what you sell, to whom, in which industries or countries. A second sentence for what you do not do.

Useful facts: founding year, headquarters, site languages. No confidential data.

## Offers
- [Offer A](https://www.example.com/offers/offer-a): which team it is for, which problem, which deliverable
- [Offer B](https://www.example.com/offers/offer-b): which team it is for, which problem, which deliverable

## Proof
- [Case studies](https://www.example.com/case-studies): results published with client approval
- [Partners and certifications](https://www.example.com/partners): what can be verified

## Resources
- [Guide to topic X](https://www.example.com/blog/topic-x): the question it answers

## Company
- [About](https://www.example.com/about): team, history, locations
- [Contact](https://www.example.com/contact): how to reach the sales team

## Optional
- [Legal notice](https://www.example.com/legal): legal information

llms.txt best practices, maintenance cost and risks

• One H1, one blockquote summary, link lists: the llmstxt.org format and nothing more.
• Canonical URLs that return 200, with no private, duplicate or fast-⁠aging pages (Semrush).
• No sensitive data: no negotiated pricing, no client names without approval, no staging or admin URLs, no personal contacts. The file is public.
• The same text your visitors see: Olivier Duffez, quoted by Adimeo, raises a cloaking risk.
• Place it at the root, and link to it from the site if you want agents to find it (Ahrefs).

Generator or hand-⁠written? Plugins such as Yoast and AIOSEO generate it, per llmstxt.org. SeoMix advises against turning these options on, while Ahrefs finds defaults reasonable if the effort stays minimal. For 20 links, hand-⁠writing is faster than reviewing. And llms-full.txt, all the text in one file, is for documentation, not for a sales site.

• Information leak: Why: The file is public and can expose structure, content gaps and messaging to competitors (Search Engine Land, Ahrefs); Safeguard: Only public links, reviewed by the site owner
• Stale file: Why: Broken links, retired offers; Search Engine Land suggests a review every few months; Safeguard: Quarterly review: every link returns 200, offers are current
• Unauthorized edit: Why: Ahrefs recommends version control, restricted write access and monitoring; Safeguard: Versioned file, restricted access, change alert
• Server error: Why: Lighthouse treats a 404 as "not applicable" but fails the audit on a server error; Safeguard: Test the URL: 200 or a clean 404, never 5xx
• Wasted time: Why: Easy to create, hard to justify maintaining, notes Search Engine Journal; Ahrefs concludes the cons outweigh the pros for AI visibility; Safeguard: Cap the effort: 1 to 2 hours to create, 30 minutes per quarter (Klickbee estimate)

Should you create an llms.txt? Our decision tree

• You publish technical docs, an API or developer guides: Our call: Yes; Why: That is the observed use: Microsoft, coding agents
• B2B marketing site where robots.txt, rendering or entity pages are unverified: Our call: Not yet; Why: A suggestion file fixes neither a block nor unreadable pages
• Foundations and logs in place: Our call: Optional; Why: 1 hour, minimal template, 90-day check; effect not proven
• Your CMS generates one by default: Our call: Review before publishing; Why: The output can list useless or sensitive pages
• A vendor sells it as a citation lever: Our call: Ask for proof; Why: No study we read shows it: ask for a log measurement

Method

Data pulled on October 26, 2026. Volumes and difficulty: DataForSEO, September 2026 (Google Ads data for the US). Sources: official documentation, measurements from Ahrefs, Semrush and SeoMix, and analyses from Search Engine Land, Search Engine Journal and Adimeo, all opened.

Limits: we did not run a log test ourselves. The Vercel study dates from December 2024, the OtterlyAI figure is relayed by Search Engine Land, and Mintlify sells llms.txt tooling. What is ours: the field findings from our audits and the decision rules, which are opinions.

Conclusion: an llms.txt file is an option, not a strategy

Should you create an llms.txt? Not first, and not to hope for citations. Repair AI crawler access, rendering, your company pages and your measurement. Then, if you want the file, keep it short, public and free of sensitive data, and judge it on your logs after 90 days. For technical documentation, create it without hesitation. In our GEO offer, llms.txt sits in the technical base, but we handle it after crawler access, rendering and company pages.

Start with what really moves your AI visibility. A free action plan ranks your repairs: crawler access, rendering, entity pages, measurement. Get your free action plan

Frequently asked questions

What is an llms.txt file?

It is a Markdown file at the root of a website that summarizes its key pages for an AI system, proposed in September 2024. It is voluntary, and no major AI company has confirmed using it for search.

Is llms.txt actually used?

By coding agents and documentation tools, yes. For search and citations, nothing shows it. Google Search says it ignores the file, and in Ahrefs' study 97% of valid files received no requests in May 2026.

Is llms.txt mandatory?

No. Lighthouse calls it optional: a 404 is "not applicable," and only a server error fails the audit.

Should I add llms.txt to my website?

Only after the foundations are fixed: crawler access, rendering, clear company pages and log measurement. Then it is a low-⁠cost option, about an hour of work and a quarterly review, with no promise of results.

Does llms.txt help SEO or Google rankings?

No. Google's guide says Google Search ignores these files, so they neither help nor harm. robots.txt, not llms.txt, controls crawler access.

Sources

• llms-⁠txt: The /llms.txt file, v2
• We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read (Ahrefs)
• AI optimization guide (Google Search Central)
• llms.txt | Lighthouse (Chrome for Developers)
• Overview of OpenAI Crawlers (OpenAI)
• What is llms.txt & should you use it? (Semrush)