Keyword Prominence Checker
Check where a keyword appears on a page, not just how often.
llms.txt Generator
Write the llms.txt file that tells AI models what your site is.
Start a section with ##. Each link is Title | URL | why it matters. A section named Optional is treated as skippable, which the format defines.
AI crawler policy — this part is real and honoured today
llms.txt
Your generated code appears here.
Quick Tool Overview
The llms.txt Generator writes the proposed AI-facing description file — a heading, a summary and curated sections of annotated links — and, more usefully, the robots.txt rules for every AI crawler that publishes a user agent. It is explicit that llms.txt is a 2024 proposal nobody has committed to reading, while the crawler rules work today.
- What it does
- Generates an llms.txt file and AI crawler robots.txt rules
- You give it
- A site description and the pages that matter most
- You get back
- An llms.txt file plus directives for GPTBot, ClaudeBot and the rest
- Best for
- Documentation sites, and anyone deciding an AI crawling policy
About the llms.txt Generator
Fill in what your site is and which pages matter, and the generator writes an llms.txt file in the proposed format: a heading, a summary, and curated sections of links with a sentence explaining each one. The AI crawler rules for robots.txt are produced alongside it.
Be clear about the status of this. llms.txt is a proposal from 2024, not a standard, and no major AI company has committed to reading it. What it costs is ten minutes and one file; what it might buy is a curated, plain-language description of your site sitting at a predictable address for whatever comes next. The second half of this tool is the part that definitely works today: the robots.txt directives for GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest are real and honoured, and they are how you actually control whether AI systems crawl you.
Key features of the llms.txt Generator
The proposed format
Heading, blockquote summary, then sections of links each with a sentence of context.
Real crawler rules
GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and more.
Training separated from search
Blocking a training crawler and blocking a search crawler are different decisions, and the tool treats them that way.
Honest about the status
It states plainly that no major AI company has committed to reading llms.txt.
Both files at once
The llms.txt content and the robots.txt block are generated together and copied separately.
How to use the llms.txt Generator
-
1
Describe your site
The name, a one-line summary, and a short paragraph of context.
-
2
List the pages that matter
Grouped into sections, one per line as "Title | URL | why it matters".
-
3
Set your AI crawler policy
Allow or block each AI crawler, and the robots.txt rules are written to match.
-
4
Publish both files
llms.txt at your domain root, and the rules appended to your existing robots.txt.
Frequently asked questions
Not verifiably. It is a proposal from 2024 and no major AI company has publicly committed to reading it. It is cheap to add and harmless, and it may matter later, but anyone telling you it improves your AI visibility today is guessing.
robots.txt controls access and is universally honoured. llms.txt describes content and is honoured by nobody in particular. If you want to stop an AI crawler, robots.txt is the file that does it — llms.txt has no mechanism for blocking anything.
The ones that publish a user agent and honour robots.txt: GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, Google-Extended, Applebot-Extended, Bytespider, CCBot and Amazonbot. This tool writes rules for each. Crawlers that ignore robots.txt are not addressable this way at all.
It depends on what you want. Blocking training crawlers keeps your content out of future models. Blocking search crawlers like OAI-SearchBot also removes you from AI answers that cite sources, which for most sites is traffic. They are separate crawlers and worth separate decisions.
Reviews of the llms.txt Generator
What people who use the llms.txt Generator think of it. Reviews come from visitors and are posted as written.
No ratings yet
Yours would be the first.
- 5 0 reviews rated 5 stars
- 4 0 reviews rated 4 stars
- 3 0 reviews rated 3 stars
- 2 0 reviews rated 2 stars
- 1 0 reviews rated 1 star
0 reviews
Newest first
No reviews yet
Be the first to rate the llms.txt Generator. It takes half a minute and no account.
Write the first reviewMore SEO tools
Keyword Grouper
Cluster a keyword list into groups by the words they share.
Meta Tag Analyzer
Paste a page and get every meta tag audited in one report.
Blog Title Generator
Turn a topic into dozens of workable title options.
LSI Keyword Extractor
Pull the related phrases a piece of content actually uses.
Meta Tag Generator
Build a complete set of meta, Open Graph and Twitter card tags.
Image Sitemap Generator
Build an image sitemap so Google can find pictures it would miss.
Twitter Card Generator
Build the twitter:card tags and preview the result.