Robots.txt Generator
Build a valid robots.txt for your website without memorising the syntax.
How to use the Robots.txt Generator
- 1Set the user-agent and paths.
- 2Add your sitemap URL.
- 3Save the output as robots.txt in your site root.
Robots.txt Generator FAQ
No. It asks well-behaved crawlers not to crawl. Use authentication or noindex to keep pages private.
At the root of the domain, for example https://example.com/robots.txt.