robots.txt: understand crawling, indexing and mistakes to avoid

This guide is part of Master technical SEO for a website. It focuses on practical decisions that make public pages easier to discover, interpret and maintain without pretending that a technical setting can guarantee visibility.

What robots.txt actually does

robots.txt controls crawler access to paths for compliant bots. It is not a reliable way to remove a URL from an index and it is not a security boundary. Keep sensitive information behind real access controls and use appropriate indexing directives when a page must not appear in search.

Why technical SEO matters

Technical SEO removes unnecessary ambiguity between a site and a search engine. It helps crawlers discover durable URLs, understand which version is canonical and interpret the page consistently. It does not guarantee a ranking: useful content, intent and site context still matter.

Crawling, indexing and ranking

Crawling is discovery and retrieval. Indexing is the decision to understand and retain a document. Ranking is the selection of indexed documents for a query. Diagnose these stages separately: a ranking problem is not automatically a crawl problem.

Keep signals coherent

The URL, canonical, title, robots directives, structured data and internal links should describe the same page. Contradictory signals force search engines to resolve conflicts. Prefer one stable public URL and link to it consistently.

Use Bethemesh tools as checks

Generators help produce valid syntax, but they do not decide strategy for you. Review every generated value in the context of the real page and validate the final public URL.

Measure after publishing

After publication, verify the rendered page, status code, canonical target, internal links and structured data. Recheck after migrations or template changes because technical regressions often appear outside the article itself.

Final check

Use the related Bethemesh tools to prepare or validate individual elements, then review the complete page as a system. The strongest technical setup is the one that stays understandable for users, maintainers and crawlers at the same time.