Does robots.txt hide pages from everyone?

No. It only asks compliant crawlers not to fetch listed paths. Malicious bots ignore it, and the file itself is public — so never list secret URLs in it.

Where do I put the generated file?

Save it as plain text named robots.txt at your domain root (for example https://example.com/robots.txt) so crawlers find it automatically.

Do all search engines respect Crawl-delay?

No. Some crawlers honor it, but Google ignores Crawl-delay — use Search Console crawl settings or server-side rate limiting for Googlebot instead.

Robots.txt Generator

Build a syntactically valid robots.txt that tells crawlers which paths they may fetch, plus sitemap and crawl-delay directives.

Generate a valid robots.txt file instantly. Free online robots.txt generator with user-agent rules, disallow paths, sitemap and crawl-delay.

About Robots.txt

Build a syntactically valid robots.txt that tells crawlers which paths they may fetch, plus sitemap and crawl-delay directives.

Every crawler reads /robots.txt before fetching your site. This generator turns your rules into a clean file: one User-agent group, a Disallow line per blocked path (empty means everything is allowed), optional Allow exceptions, an optional Crawl-delay in seconds, and your sitemap URL.

Remember that robots.txt is advisory — well-behaved crawlers obey it, but it does not hide private content. Never rely on it for secrets; use authentication instead.

The file is generated entirely in your browser; paste the result at your domain root as robots.txt.

Where it is used

Launching a new site, blocking admin/staging areas from indexing, declaring the sitemap location, and slowing aggressive crawlers.

How to use the Robots.txt

User-agent — enter the value (e.g. *).
Disallow Paths (one per line) — enter the value (e.g. /admin/ /private/ /tmp/).
Allow Paths (one per line, optional) — enter the value (e.g. /public/).
Sitemap URL (optional) — enter the value (e.g. https://example.com/sitemap.xml).
Crawl-delay in Seconds (optional) — enter the value (e.g. 10).
Calculate — press the Calculate button to see the result instantly above.

Formula

One directive per line: User-agent, Disallow/Allow paths, optional Crawl-delay and Sitemap.

Examples

Example

userAgent:*

disallow:/admin/ /private/

allowPaths:

sitemapUrl:https://example.com/sitemap.xml

crawlDelay:

Result: User-agent: * / Disallow: /admin/, /private / Sitemap declared

Frequently Asked Questions

Does robots.txt hide pages from everyone?: No. It only asks compliant crawlers not to fetch listed paths. Malicious bots ignore it, and the file itself is public — so never list secret URLs in it.
Where do I put the generated file?: Save it as plain text named robots.txt at your domain root (for example https://example.com/robots.txt) so crawlers find it automatically.
Do all search engines respect Crawl-delay?: No. Some crawlers honor it, but Google ignores Crawl-delay — use Search Console crawl settings or server-side rate limiting for Googlebot instead.

Calculator Inputs