Skip to content
ToolQivo

Robots.txt Tester

Check which URLs your robots.txt actually allows and blocks.

SEO Tools Reviews Free · no sign-up

A crawler reads only the one group that matches it best, and ignores every other group entirely.

Quick Tool Overview

The Robots.txt Tester matches your URLs against your rules the way a crawler does: longest matching path wins, Allow beats Disallow at equal length, wildcards and dollar anchors are honoured, and only the single best-matching user-agent group applies. Each URL shows the exact line that decided it, not just the verdict.

What it does
Tests which URLs a robots.txt allows and blocks
You give it
Your robots.txt and a list of URLs to check
You get back
A verdict per URL and the rule that produced it
Best for
Debugging a block, and checking a change before it ships
SEO Tools Free, unlimited use No sign-up Runs in your browser

About the Robots.txt Tester

Paste your robots.txt, list the URLs you want to check, and each one is matched against the rules the way a crawler matches them — longest matching rule wins, allow beats disallow at equal length, and wildcards and dollar anchors are honoured.

The reason a tester is worth having is that robots.txt matching is not intuitive. The rules are prefix matches, not path segments, so <span>Disallow: /admin</span> also blocks /administrator and /admin-guide. A crawler only reads the one user-agent group that matches it best and ignores every other group entirely, which is why a rule under the wildcard group can appear to do nothing once a Googlebot group exists. And the specificity contest is decided by the character length of the matching path, not by the order the lines appear in. This tool shows you which rule won for each URL, not just the verdict.

Key features of the Robots.txt Tester

Real matching rules

Longest match wins and Allow beats Disallow at equal length — line order is irrelevant, exactly as Google treats it.

Wildcards and anchors

Asterisks and trailing dollar signs are interpreted properly rather than compared as plain text.

Per-crawler groups

Pick a user agent and only the group that applies to it is used, which is where most surprises come from.

Syntax errors flagged

Rules outside a group, unknown directives and misspelt fields are reported with their line numbers.

Explains the verdict

Each result names the winning rule, so a surprising block is traceable in one step.

How to use the Robots.txt Tester

  1. 1

    Paste your robots.txt

    The whole file. Fetch it from yourdomain.com/robots.txt and copy it in.

  2. 2

    List the URLs to test

    One per line — full URLs or paths. Both work.

  3. 3

    Choose a crawler

    Googlebot, Bingbot or any other agent, so the right group of rules is applied.

  4. 4

    Read the verdicts

    Each URL shows allowed or blocked, and the exact line that decided it.

Frequently asked questions

No, and this is the most consequential misunderstanding about it. It stops a crawler fetching the page. A blocked URL can still be indexed from links pointing at it, appearing with no description at all. To keep a page out of the index, allow crawling and use a noindex directive.

Reviews of the Robots.txt Tester

What people who use the Robots.txt Tester think of it. Reviews come from visitors and are posted as written.

No ratings yet

Yours would be the first.

  • 5 0 reviews rated 5 stars
  • 4 0 reviews rated 4 stars
  • 3 0 reviews rated 3 stars
  • 2 0 reviews rated 2 stars
  • 1 0 reviews rated 1 star

Write a review

Used this tool? Your rating helps the next person decide.

Your rating

Plain text only — links, email addresses and HTML are not accepted. Your name and review are shown publicly; nothing else is collected. See the privacy policy.

0 reviews

Newest first

No reviews yet

Be the first to rate the Robots.txt Tester. It takes half a minute and no account.

Write the first review

More SEO tools