Skip to main content
Bethemesh
Web & SEO

robots.txt generator

Configure Allow and Disallow rules, sitemaps, host and crawl delay, then copy a robots.txt file ready to publish.

  • Clipboard

Embed this widget

Customize the result, check the live preview, then copy the code.

Preview

Embed code type

Code to copy

Responsive and automatically resized by Bethemesh.

Preset

Use * to apply the rules to every crawler.

One path per line, for example /public/.

One path per line, for example /admin/.

One absolute URL per line.

Optional, in seconds. Not every search engine supports it.

Optional directive recognized mainly by some search engines.

Generated file

Was this tool useful?

How does this tool work?

This generator writes a robots.txt file from a few fields: target crawler, allowed and disallowed paths, crawl delay, sitemaps and preferred host. It offers ready-made presets, normalises paths and warns you before producing a rule that would block the whole site.

The file starts with User-agent, followed by the Allow lines and then the Disallow lines. A path entered without a leading slash gets one (admin/ becomes /admin/) and duplicates are removed.

The crawl delay is written only if it is zero or a positive number; sitemap URLs must be absolute http or https addresses, otherwise a warning appears.

The preferred host is reduced to its domain name (https://www.example.com/ becomes www.example.com) and added in a Host line.

Presets fill in the fields: Allow the whole site (Allow: /), Block the whole site (Disallow: /) or Block private areas (/admin/, /account/, /api/private/).

How to use this tool

  1. Choose a preset

    Or start from Custom to enter everything yourself.

  2. Enter the target crawler

    * for all crawlers, or a name such as Googlebot or GPTBot.

  3. List the paths

    One path per line in Allowed paths and Disallowed paths, then add the sitemap URLs.

  4. Generate and publish

    Copy the generated file and place it at the root of the domain: https://example.com/robots.txt.

Use cases

Launch a new site

Publish a minimal robots.txt that declares the sitemap and lets everything be crawled.

Keep crawlers out of low-value areas

Stop crawlers spending time on the admin area, customer account or internal search results.

Manage AI crawlers

Prepare a block for a specific crawler, for example to refuse crawling by an AI training bot.

Examples

Private areas blocked with one exception

Allow reopens the API documentation inside a disallowed directory: at Google, the most specific rule wins. The sitemap is declared at the end of the file.

Input

Preset: Block private areas
Target crawler: *
Allowed paths: /api/private/docs/
Disallowed paths: /admin/, /account/, /api/private/
Sitemap URLs: https://example.com/sitemap.xml

Result

Generated file:
User-agent: *
Allow: /api/private/docs/
Disallow: /admin/
Disallow: /account/
Disallow: /api/private/

Sitemap: https://example.com/sitemap.xml

Block an AI crawler

Only GPTBot gets Disallow: /; the tool warns that this crawler will not crawl any page. Other crawlers are not affected by this block.

Input

Target crawler: GPTBot
Disallowed paths: /

Result

Warning: Disallow: / asks the targeted crawlers not to crawl any page.
User-agent: GPTBot
Disallow: /

Tips and best practices

  • robots.txt blocks crawling, not indexing: a disallowed page can still appear in Google if it has links pointing to it. To keep it out, use noindex and leave it crawlable.
  • The file is not a security measure: it is public and only well-behaved crawlers follow it. Protect sensitive areas with authentication.
  • Google ignores Crawl-delay; Bing and some other engines honour it. Host is no longer used by the major search engines.
  • Do not block CSS and JavaScript files: Google needs them to render the page as a user would.

Sources and references

Frequently asked questions

Where should the robots.txt file go?

At the root of the host, exactly at /robots.txt. Each subdomain and protocol has its own: blog.example.com/robots.txt does not apply to example.com.

What happens if Allow and Disallow conflict?

Under RFC 9309, which Google follows, the rule with the longest path wins; on a tie, Allow wins. That is how you reopen a subfolder inside a disallowed directory.

How do I target several crawlers?

The tool generates a block for one crawler at a time. Generate one block per crawler and combine them in the same file, separated by a blank line.

Are * and $ allowed in paths?

Yes, the tool copies them as they are. Google understands them: /*.pdf$ blocks every URL ending in .pdf. Check that the other search engines you target support them too.

Is the file sent to a server?

No. It is generated in your browser; uploading it to your site is up to you.

Related tools

Keep going with these tools

Tools similar or complementary to this one.

Similar tools

XML sitemap diff

Compare two sitemaps or URL lists and identify added and removed URLs.

Web & SEOUse this tool

Meta tags generator

Create SEO, Open Graph and Twitter/X tags for a web page.

Web & SEOFeaturedUse this tool

Read next