Skip to content
ToolQivo

AI Crawler Blocker

Generate robots.txt rules to block GPTBot, ClaudeBot, CCBot and other AI crawlers.

AI Tools Reviews Free · no sign-up
Quick choice

Quick Tool Overview

The AI Crawler Blocker generates robots.txt rules for more than twenty-five AI user agents grouped into training, AI search and user-triggered fetchers, so you can block model training while deciding separately whether AI search can still cite you.

What it does
Generates robots.txt rules for AI crawlers
You give it
The crawlers to block and the paths to protect
You get back
A complete robots.txt, merged with your own
Best for
Publishers, bloggers and site owners
AI Tools Free, unlimited use No sign-up Runs in your browser

About the AI Crawler Blocker

AI companies crawl the web for three different reasons, and robots.txt snippets copied from blog posts usually treat them as one. Training crawlers such as GPTBot, ClaudeBot, CCBot and Google-Extended collect pages to train models. AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so answers can cite and link to you. User-triggered fetchers such as ChatGPT-User visit when a person asks an assistant to read a page.

This generator lists more than twenty-five current AI user agents grouped by purpose, with the company behind each and what blocking it actually costs. Block training while keeping AI search open in one click, block everything, or pick individually. Block the whole site or only specific paths, lay the rules out as one group per purpose or one block per crawler, and add your sitemap.

Paste your existing robots.txt and the AI rules are added to it, with your sitemap lines kept at the end and a warning if a crawler already has a group of its own. The notes are candid about the limits: robots.txt is a request that the major AI companies honour, not a lock, and blocking Google-Extended does not affect Google Search.

Key features of the AI Crawler Blocker

Grouped by purpose

Training crawlers, AI search crawlers and user fetchers are listed separately with their cost.

One-click policies

Block training only, block every AI crawler, or clear all.

Whole site or specific paths

Protect everything, or just sections such as /blog/.

Merges with your robots.txt

Existing rules and sitemap lines are kept, and duplicate groups are flagged.

Current user agents

GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot, CCBot and more.

How to use the AI Crawler Blocker

  1. 1

    Choose what to block

    Block training only, block every AI crawler, or tick individual user agents.

  2. 2

    Set the scope

    The whole site, or only paths such as /blog/ or /research/.

  3. 3

    Merge with your file

    Paste your current robots.txt so existing rules and sitemaps are kept.

  4. 4

    Upload robots.txt

    Save the result as robots.txt in the root of your domain.

Frequently asked questions

Add a group with User-agent: GPTBot and Disallow: / to your robots.txt to stop training. OAI-SearchBot controls whether ChatGPT search can show your pages, and ChatGPT-User covers pages a user asks it to open.

Reviews of the AI Crawler Blocker

What people who use the AI Crawler Blocker think of it. Reviews come from visitors and are posted as written.

No ratings yet

Yours would be the first.

  • 5 0 reviews rated 5 stars
  • 4 0 reviews rated 4 stars
  • 3 0 reviews rated 3 stars
  • 2 0 reviews rated 2 stars
  • 1 0 reviews rated 1 star

Write a review

Used this tool? Your rating helps the next person decide.

Your rating

Plain text only — links, email addresses and HTML are not accepted. Your name and review are shown publicly; nothing else is collected. See the privacy policy.

0 reviews

Newest first

No reviews yet

Be the first to rate the AI Crawler Blocker. It takes half a minute and no account.

Write the first review

More AI tools