Robots.txt Tester

A robots.txt file is deceptively simple to read and surprisingly easy to get wrong: a stray `Disallow: /` under the wrong User-agent group can silently de-index an entire site, while a rule that looks like it blocks a crawler can be overridden by a longer, more specific pattern elsewhere in the file. This tool reimplements the actual matching algorithm search engines and AI crawlers use — the de-facto Robots Exclusion Protocol formalized in RFC 9309 and documented by Google — rather than approximating it with a simple line-by-line scan, so the verdict you see here is the verdict a real crawler would compute.

Paste your robots.txt content, type the URL path you want to check (like `/admin/` or `/blog/draft-post`), and pick a crawler from a bundled list of 20 real user-agent tokens. That list intentionally goes beyond the classics — Googlebot, Bingbot, and the `*` fallback — to include the current generation of AI scrapers: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot and anthropic-ai (Anthropic), Google-Extended (Google's AI-training opt-out signal, separate from Googlebot's search crawler), Applebot-Extended (Apple's equivalent opt-out), PerplexityBot, CCBot (Common Crawl, whose archive feeds many other AI models), Bytespider (ByteDance), Meta-ExternalAgent, Amazonbot and more. Every token ships with a short, honest explanation of what it actually is.

The algorithm itself follows the spec precisely: user-agent groups are matched case-insensitively, with an exact or prefix match on the product token always beating the `*` fallback group, and multiple blocks declaring the same token are merged as real crawlers do. Within the winning group, every Allow and Disallow rule is checked against the path using proper wildcard semantics (`*` matches any sequence, a trailing `$` anchors to the end of the path), and the longest matching pattern wins — with an Allow winning any tie against a Disallow of equal length, exactly as documented by Google's own robots.txt implementation.

The result panel doesn't just say ALLOWED or BLOCKED — it names the exact line and rule text responsible, so you can go straight back to your file and fix (or confirm) the directive that matters. That is the detail most robots.txt checkers skip, and it is the difference between guessing why a page isn't being crawled and knowing for certain.