This guide is part of Master technical SEO for a website. It focuses on practical decisions that make public pages easier to discover, interpret and maintain without pretending that a technical setting can guarantee visibility.
GuideBest practicesBeginner
robots.txt: understand crawling, indexing and mistakes to avoid
Use robots.txt correctly, understand what it can and cannot block, and avoid confusing crawl control with deindexing.
Show contents
What robots.txt actually does
robots.txt controls crawler access to paths for compliant bots. It is not a reliable way to remove a URL from an index and it is not a security boundary. Keep sensitive information behind real access controls and use appropriate indexing directives when a page must not appear in search.
Why technical SEO matters
Technical SEO removes unnecessary ambiguity between a site and a search engine. It helps crawlers discover durable URLs, understand which version is canonical and interpret the page consistently. It does not guarantee a ranking: useful content, intent and site context still matter.
Crawling, indexing and ranking
Crawling is discovery and retrieval. Indexing is the decision to understand and retain a document. Ranking is the selection of indexed documents for a query. Diagnose these stages separately: a ranking problem is not automatically a crawl problem.
Keep signals coherent
The URL, canonical, title, robots directives, structured data and internal links should describe the same page. Contradictory signals force search engines to resolve conflicts. Prefer one stable public URL and link to it consistently.
Use Bethemesh tools as checks
Generators help produce valid syntax, but they do not decide strategy for you. Review every generated value in the context of the real page and validate the final public URL.
Measure after publishing
After publication, verify the rendered page, status code, canonical target, internal links and structured data. Recheck after migrations or template changes because technical regressions often appear outside the article itself.
Final check
Use the related Bethemesh tools to prepare or validate individual elements, then review the complete page as a system. The strongest technical setup is the one that stays understandable for users, maintainers and crawlers at the same time.