Robots.txt Generator
Control how search engine bots crawl and index your website.
About the robots.txt Generator
The robots.txt Generator builds a valid robots.txt file that tells search engines and AI crawlers which parts of your site they can and cannot access. Choose which bots to allow or block, add specific disallow paths, and link your sitemap, all through a simple form.
What it does
The tool assembles a properly formatted robots.txt with User-agent, Allow, Disallow and Sitemap directives. It supports common bots such as Googlebot, Bingbot, GPTBot, ClaudeBot and PerplexityBot, and warns you about rules that could accidentally block your entire site.
Why it matters
A single wrong line in robots.txt can wipe your site from Google overnight. Getting the syntax right, and knowing which crawlers to allow or block, is critical for SEO, AI visibility, security and server performance.
How to use it
- Decide which sections of your site should stay private (admin, cart, staging).
- Add each private path as a Disallow line.
- Add your sitemap URL so search engines can find your pages.
- Choose whether to allow or block AI crawlers such as GPTBot.
- Copy the output and upload it to the root of your domain as /robots.txt.
Best-practice tips
- Never Disallow: / on your production site unless you intend to hide everything.
- Do not rely on robots.txt for security; use authentication for private data.
- Always link your XML sitemap in robots.txt.
- Test the final file with Google Search Console's robots.txt tester.
- Update robots.txt when you launch new sections, staging environments or campaigns.
Frequently asked questions
Where does robots.txt live?
It must be at the root of your domain, e.g. https://yourdomain.com/robots.txt. Subfolders are ignored.
Should I block AI crawlers?
Only if you do not want your content used to train or answer LLM queries. Blocking them can reduce visibility in ChatGPT, Perplexity and AI Overviews.
Is robots.txt case-sensitive?
Yes. Paths are case-sensitive, so /Blog/ and /blog/ are treated as different URLs.