robots.txt generator

Create a clear robots.txt file to guide web crawlers.

Preset

Target crawler

Use * to apply the rules to every crawler.

Allowed paths

One path per line, for example /public/.

Disallowed paths

One path per line, for example /admin/.

Sitemap URLs

One absolute URL per line.

Crawl delay

Optional, in seconds. Not every search engine supports it.

Preferred host

Optional directive recognized mainly by some search engines.