Robots.txt Tester
Verify USA site crawl blocks before launch in seconds. Paste robots.txt, test path like /private/page.html, get Blocked/Allowed plus rule instantly offline with no signup plus subtree-versus-exact semantics and crawl-not-index warnings for SEOs devs shops bloggers and agencies across pre-launch QA workflows daily plus auth-layering advice with slash-variant tests for SEOs devs shops bloggers SaaS and agencies across crawl-gate reviews daily.
Use this Robots.txt Tester to check your Allow and Disallow rules against any URL path and see whether crawlers are blocked.
Calculator Inputs
About Robots Tester
Verify USA site crawl blocks before launch in seconds. Paste robots.txt, test path like /private/page.html, get Blocked/Allowed plus rule instantly offline with no signup plus subtree-versus-exact semantics and crawl-not-index warnings for SEOs devs shops bloggers and agencies across pre-launch QA workflows daily plus auth-layering advice with slash-variant tests for SEOs devs shops bloggers SaaS and agencies across crawl-gate reviews daily.
Last matching Allow/Disallow wins. Empty path allows all. Case path-sensitive.
Block: /private/ blocks subtree. /private exact only file.
Crawl vs index: Robots stops crawl not index — noindex for hide.
Privacy: Local.
Limits: * agent only, no $ wildcards full.
Admin blocks: Austin SEOs blocking $500,000 client admin panels test exact paths pre-launch. New York agencies QAing $300,000 launches verify staging stays hidden on 09/22/2026.
Staging guards: Chicago devs guarding $2M pre-release sites confirm wildcard blocks catch every route. Seattle SaaS shielding $9M app interiors layers auth behind robots.
Cart flows: Miami shops hiding $800,000 checkout flows keep thin pages out of indexes. Denver bloggers keeping $1,000 drafts private test before scheduling.
Crawl versus index: Robots stops fetching, never removes indexed URLs — noindex tags hide, disallow merely skips.
Match order: Last matching rule wins; empty paths allow all. Test both trailing-slash variants explicitly.
Panel blocks: Austin SEOs blocking $500,000 client admin panels test exact paths pre-launch. Boston hospitals shielding $40M patient portals layer auth behind robots on 09/22/2026.
Staging guards: Chicago devs guarding $2M pre-release sites confirm wildcards catch every route. Phoenix schools hiding $200,000 student records verify privacy.
Flow hiding: Miami shops hiding $800,000 checkout flows keep thin pages unindexed. Denver bloggers keeping $1,000 drafts private test before scheduling.
Interior shields: Seattle SaaS shielding $9M app interiors never relies on robots alone. New York agencies QAing $300,000 launches verify staging invisibility.
Slash discipline: Test both trailing-slash variants — subtree versus exact semantics hinge on one character.
Where it is used
SEOs blocking /admin, devs staging no-crawl, shops hiding cart, bloggers private drafts, and agencies pre-launch QA Austin SEOs block $500,000 admin panels, Chicago devs guard $2M staging sites, Miami shops hide $800,000 cart flows, Denver bloggers keep $1,000 drafts private, Seattle SaaS shields $9M app interiors, and New York agencies QA $300,000 launches Austin SEOs block $500,000 admin panels, Chicago devs guard $2M staging sites, Miami shops hide $800,000 cart flows, Denver bloggers keep $1,000 drafts private, Seattle SaaS shields $9M app interiors, New York agencies QA $300,000 launches, Boston hospitals shield $40M patient portals, and Phoenix schools hide $200,000 student records.
How to use the Robots Tester
- Robots.txt Content — enter the value (e.g. User-agent: * Disallow: /private/).
- Test URL Path — enter the value (e.g. /private/page.html).
- Calculate — press the Calculate button to see the result instantly above.
Formula
parse Allow/Disallow lines ignore # → if url startsWith path last match wins blocked=Disallow.
Examples
Private block
Public page
Allow override
Frequently Asked Questions
- Crawl vs index?
- Robots blocks crawl not index. Noindex meta hides from results.
- Allow override?
- Yes last match wins. Allow after Disallow opens path.
- Case?
- Paths case-sensitive. /Private ≠ /private.
- Sitemap line?
- Ignored for block test. Keep Sitemap: separate.
- Test live?
- Paste live robots.txt + Search Console tester confirm.
- Private?
- Yes. Browser-local.