Loading page. Please wait.
Paste robots.txt and test a URL for Googlebot or another crawler token. See the matching group, winning rule and line number. No live crawling or account required.
Runs in your browser. No account, uploads, live scan or provider charge. Input values are not sent to Mapseen. With analytics consent, we measure tool actions and whether you used the example.
Paste rules from the same origin as your URL, or use the fictional example. An empty file allows crawling by default.
All URLs are fictional. These fixtures exercise the same parser as the tool, using its Google-style rule matching. Other crawlers may implement different behaviour. No live URL is fetched.
User-agent: *
Disallow: /private/
User-agent: Googlebot
Disallow: /drafts/Test: https://example.com/private/page · Googlebot
Result: Allowed · No blocking rule.
Googlebot uses its specific group here. The wildcard group's /private/ rule is not inherited.
User-agent: Googlebot
Disallow: /private/
User-agent: Googlebot
Allow: /private/helpTest: https://example.com/private/help · Googlebot
Result: Allowed · Winning line 5.
Both matching groups contribute rules. The longer /private/help rule wins.
User-agent: *
Disallow: /same
Allow: /sameTest: https://example.com/same · Googlebot
Result: Allowed · Winning line 3.
Both rules match the same path length. Allow resolves the tie.
User-agent: *
Disallow: /*.pdf$Test: https://example.com/guide.pdf · Googlebot
Result: Blocked · Winning line 2.
The wildcard matches any path prefix and $ anchors the end. A query string would change the tested path-and-query.
User-agent: *
Disallow: /Private/Test: https://example.com/private/page · Googlebot
Result: Allowed · No blocking rule.
/Private/ does not match /private/. Test the actual URL spelling.
User-agent: *
Noindex: /public/Test: https://example.com/public/page · Googlebot
Result: Allowed · No blocking rule.
The checker ignores and warns about the unsupported line. Page-level or HTTP-header noindex is a separate check.
Line 2: noindex does not determine Google crawl access here. In particular, noindex is not supported in robots.txt.
Paste the robots.txt content served by the same origin as the URL you want to test. The checker applies exact crawler product-token selection and common Google path matching locally. It shows the winning rule and warns about ignored or unsupported lines. It never fetches the website.
Not reliably. A blocked URL can still appear in results. Google needs crawl access to see a noindex directive, while authentication protects private content.
Groups matching the entered product token are combined; the wildcard group is used when no exact group exists. Within it, the longest matching path wins; an equivalent Allow takes precedence over Disallow.