robots.txt erstellen

Erstellt eine robots.txt-Datei mit Regeln für Suchmaschinen-Crawler und wichtige Bots.

Preset

Ziel crawler

Verwende *, um die Regeln auf alle Crawler anzuwenden.

Allowed paths

One path per line, for example /public/.

Disallowed paths

One path per line, for example /admin/.

Sitemap URLs

One absolute URL per line.

Crawl delay

Optional, in seconds. Not every search engine supports it.

Preferred host

Optional directive recognized mainly by some search engines.