100% local and secure
Your data stays on your device and is never sent to our servers.
robots.txt generator
Configure Allow and Disallow rules, sitemaps, host and crawl delay, then copy a robots.txt file ready to publish.
- 100% local
- Instant
- Free
- No account
- Works offline
- Clipboard
Preset
Use * to apply the rules to every crawler.
One path per line, for example /public/.
One path per line, for example /admin/.
One absolute URL per line.
Optional, in seconds. Not every search engine supports it.
Optional directive recognized mainly by some search engines.
Generated file
Why use this tool?
Prepare a readable robots.txt file to tell web crawlers which areas they may crawl. The tool generates the text locally without sending the configuration to a server.
Smart processing
Create and verify web content with settings designed for usable results.
Supported formats
Web formats, metadata, and structured content depending on the available features.
Save time
Get a clean, ready-to-use result in seconds without installing software or configuring a complex workflow.
Guide
How to use this tool
- 1
Choose a preset
Start from a common configuration or create your own rules.
- 2
Add paths
Enter allowed and blocked areas plus sitemap URLs.
- 3
Review and copy
Read the warnings and copy the generated file to the site root.
Tips and best practices
- robots.txt controls crawling, not guaranteed de-indexing of an already known URL.
- Never expose secrets in a path and assume robots.txt makes it private.
- Use absolute URLs for Sitemap directives.
Frequently asked questions
Where should robots.txt be placed?
The file must be available at the domain root, for example https://example.com/robots.txt.
Does Disallow keep a page out of Google?
Not always. A URL may still be indexed when discovered elsewhere. Use appropriate indexing controls and real access protection for private content.
Does crawl delay work everywhere?
No. The Crawl-delay directive is not supported consistently by every search engine.
Was this tool useful?
Read next
GuideBest practicesBeginner
robots.txt: understand crawling, indexing and mistakes to avoid
Use robots.txt correctly, understand what it can and cannot block, and avoid confusing crawl control with deindexing.
29 August 20262 minRead
GuideBest practicesBeginner
Technical SEO checklist before publishing a Web page
Check metadata, canonical, robots, structured data, URL, internal links and images before a page goes live.
29 August 20262 minRead
GuideBest practicesBeginner
Technical SEO: how search engines crawl and interpret a page
Understand crawling, indexing, canonicals, metadata, structured data, performance and internal linking without confusing optimization with ranking.
29 August 20262 minRead
GuideBest practicesBeginner
How to optimize 50 images for the Web at once
Resize a batch of images, convert them to WebP and remove metadata with one reusable Pipeline.
6 September 20262 minRead
Recommended workflow
Continue your processing
Discover tools that naturally fit before, after, or alongside this one.
Complementary tools
Web & SEO
XML sitemap builder and validator
Create an XML sitemap from a list of URLs.
100% localNew
Use this toolWeb & SEO
Meta tags generator
Create SEO, Open Graph and Twitter/X tags for a web page.
100% local
Use this toolWeb & SEO
UTM URL builder
Add UTM parameters to a campaign URL.
100% localNew
Use this tool