By Team Hypomnema · Web & network
Test crawl permissions with matching rules and user-agent precedence.
Use the guide alongside the working tool.
Open Robots.txt Rule Tester →Explicit sample
User-agent: *
Disallow: /private/
Allow: /private/help$
Disallow: /*?preview=
User-agent: Googlebot
Disallow: /draft/
Sitemap: https://example.com/sitemap.xmlTest crawl permissions with matching rules and user-agent precedence.
Prepare your input
Test crawl permissions with matching rules and user-agent precedence.
Use the labeled working sample to see the accepted format. Replace it with your own input, select the relevant options, and run the tool. Supporting controls remain in the bounded editor window.
Understand the result
Local Google-style robots matching: most-specific matching user-agent groups merge, longest matching path wins, Allow wins ties, * and trailing $ are supported. UTF-8 percent-encoding is normalized with reserved octets preserved. No live crawl, robots download, indexing guarantee or support for crawl-delay. Robots exclusions are not authorization controls.
Export deliberately
Record exports contain the complete selected dataset; reports summarize up to 100 records. Artifacts contain the actual generated configuration, sanitized capture or compressed bytes. Review configuration before publishing it.
Input stays in browser memory. No remote page fetching, account access or file upload is performed.
Ready to apply this workflow?
Try Robots.txt Rule Tester →Explore a related workflow
- XML Sitemap Studio: practical guide
Build or inspect XML sitemaps with duplicate and URL checks.
- Hreflang Cluster Studio: practical guide
Generate reciprocal language alternates and audit page clusters.
- Redirect Map Auditor: practical guide
Find redirect loops, conflicts and chains before a site migration.
- Canonical URL Auditor: practical guide
Review canonical targets, chains and indexability conflicts from crawl data.
Keep your original files and review the result. Found unclear instructions? Contact Team Hypomnema.