FreeTechnical SEO tool

AI Robots.txt Generator

Tick the AI crawlers you want to let in and untick the ones you want to keep out, then click Generate snippet. You get one robots.txt group per bot, with either an open Disallow line or a site-wide block, ready to paste into your existing file.

12 AI crawlersAll allowed by defaultRuns in your browser

Updated · by the Linkstonic team

linkstonic.com/tool/ai-robots-txt-generatorLive
How it works

What the snippet contains

The output is short and deliberately simple. That simplicity has a few side effects worth knowing before you paste it.

01

The twelve bots

GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, anthropic-ai, Cohere-AI, Diffbot, Omgilibot, FacebookBot, Applebot-Extended and CCBot. Every box starts ticked, so if you generate without changing anything, every bot gets an allow.

02

Allow or block, nothing between

A ticked bot gets "Disallow:" with nothing after it, which means no path is off limits. An unticked bot gets "Disallow: /", which covers the whole site. There are no folder-level rules, Allow lines or Crawl-delay settings.

03

A snippet, not a file

The output starts with a comment line and holds only the AI groups. There is no User-agent: * group and no Sitemap line, so don't replace your robots.txt with it. Paste it into your current file, above or below the existing rules.

04

Allowing can undo your other rules

A crawler that finds a group with its own name follows that group only and ignores the * group. If your * group blocks /cart/ or /admin/, a ticked GPTBot with an empty Disallow line can now crawl those paths. Copy the important Disallow lines into the AI groups if that matters.

05

Watch the FacebookBot box

The FacebookBot checkbox writes facebookexternalhit, which is Meta's link-preview fetcher. Unticking it can stop Facebook and Messenger from showing a title and image when people share your links. Meta's AI crawler, Meta-ExternalAgent, isn't in this list.

Worked example

Blocking two bulk crawlers

A publisher who wants to stay visible in AI answers but keep bulk dataset crawlers out unticks Bytespider and CCBot. This is what the generated snippet contains.

Input
Ticked (allow): GPTBot, ClaudeBot, PerplexityBot, Google-Extended, anthropic-ai, Cohere-AI, Diffbot, Omgilibot, FacebookBot, Applebot-Extended
Unticked (block): Bytespider, CCBot
Output
12User-agent groups
10Groups with an empty Disallow:
2Groups with Disallow: /
User-agent: CCBotLast group in the file
0Sitemap or * lines

The ten allowed groups do nothing on a site whose robots.txt has no other rules. They only start to matter if you already block paths for all crawlers, because then they override that block.

AI Robots.txt Generator illustration: a robot icon next to a sample result card with user-agent groups and groups with an empty Disallow: rows
AI Robots.txt Generator, sample result
Our take

Deciding which AI bots get in

The first question isn't technical. Do you want your content in AI training data, in AI answers, both or neither? Training crawlers like GPTBot, ClaudeBot and CCBot feed future models. Answer bots like PerplexityBot fetch pages to cite them. Treat them as separate decisions, because blocking one family doesn't block the other.

Google-Extended is the one people misread. It only covers Gemini apps and grounding. AI Overviews and AI Mode run on Googlebot, so blocking Google-Extended won't take you out of them, and blocking Googlebot would take you out of Search entirely.

Keep in mind that robots.txt is a public request. The large AI companies say they follow it, but it is not access control. If content must stay private, put it behind a login. And note what this list leaves out: OAI-SearchBot and ChatGPT-User, which drive ChatGPT search visits, need their own lines if you want to control them.

Practical tips

Putting the snippet live safely

A robots.txt mistake can block far more than you meant to. These steps keep it contained.

  1. 01

    Download your current robots.txt and keep a copy before you edit anything.

  2. 02

    Paste the AI groups below your existing rules, and copy any important Disallow paths from the * group into each allowed AI group.

  3. 03

    Add groups for OAI-SearchBot, ChatGPT-User, Meta-ExternalAgent or Amazonbot by hand if you want to control them, since the generator doesn't list them.

  4. 04

    Leave the FacebookBot box ticked unless you're sure you don't need link previews on Facebook.

  5. 05

    After uploading, load yoursite.com/robots.txt in a private window to confirm the live file changed, then watch your logs for a week.

AI Robots.txt Generator tips illustration: a checklist card that starts with "Download your current robots" and "Paste the AI groups below your"
AI Robots.txt Generator, checklist
Keep going

Related free tools

All free tools
Free tool

AI Crawler Checker

A 20-bot list of AI user-agent tokens to audit against.

Open tool Free tool

LLM.txt Generator

Point language models to the pages you most want read.

Open tool Free tool

XML Sitemap Generator

Wrap a URL list in sitemap markup and download it.

Open tool
Our customers

How we help marketers win

Ttechventures
Linkstonic for our team has been a game-changer. It's helped us work more efficiently, cutting down on manual checks so we can focus on what really matters—measuring AI visibility, fixing technical SEO, and driving growth.
SK
Sarah KimHead of SEO, TechVentures
Result · techventures

+373%

Increase in share of voice

YoY on the same priority topics and tracked prompts—organic plus AI surfaces combined.

Nnorthline
Search Console data and AI visibility finally in one place. We ship the monthly readout in a fraction of the time—and execs actually understand what moved and why.
JP
James ParkVP Growth, Northline
Result · northline

−62%

Less time on monthly readouts

GSC plus AI visibility in one narrative—fewer decks and fewer “what changed?” loops before steering.

Aarclabs
We went from invisible in AI answers to consistently cited in ChatGPT and Perplexity within three months. Linkstonic showed us exactly which pages to fix and why.
PN
Priya NairContent Lead, Arclabs
Result · arclabs

+184%

Lift in branded AI mentions

Tracked on the same brand and product prompts across ChatGPT, Perplexity, and Gemini.

01techventuresShare of voice
02northlineReporting time
03arclabsAI mentions
FAQ

AI Robots.txt Generator questions

01How do I block AI crawlers with robots.txt?

Add a group for each crawler with its user-agent token and Disallow: /, for example User-agent: GPTBot on one line and Disallow: / on the next. Untick a bot in this generator and it writes that pair for you.

02Can I paste the output as my whole robots.txt?

No. It only holds AI crawler groups. It has no User-agent: * rules and no Sitemap line, so replacing your file with it would drop any existing rules for Googlebot and other crawlers. Add it to your current file instead.

03Does blocking AI bots hurt my Google rankings?

Blocking AI-specific tokens like GPTBot or Google-Extended doesn't affect Google Search, which uses Googlebot. The bigger risk is blocking an answer bot like PerplexityBot or OAI-SearchBot and losing citations in those tools.

04Do all AI crawlers obey robots.txt?

The major ones, including OpenAI, Anthropic, Google and Perplexity, say they follow it. Robots.txt is voluntary though, so a crawler that ignores it isn't stopped by it. For hard blocking, use firewall or CDN rules.

05What's the difference between ClaudeBot and anthropic-ai?

Both are Anthropic tokens. ClaudeBot is the current crawler name, and anthropic-ai is an older token that still appears in many robots.txt files. The generator writes a group for each, and keeping both costs nothing.

Beyond one-off checks

See how your pages do in Google and AI answers

Linkstonic tracks rankings, audits your site and shows whether ChatGPT, Gemini and Perplexity mention you. The free plan needs no credit card.

Start free More free tools