Robots.txt Generator free, instant.
Build a correct robots.txt file: keep crawlers out of admin, cart and search pages, link your sitemap, and choose whether to block AI training crawlers like GPTBot, ClaudeBot and Google-Extended while keeping AI search bots allowed.
Upload it to your site root so it opens at yoursite.com/robots.txt, then check it with our robots.txt Validator. robots.txt controls crawling, not indexing, and isn’t a security tool.
- Your input stays in your browser
- No limits
- No account needed
What Is a robots.txt File?
robots.txt is a plain-text file at yoursite.com/robots.txt that tells crawlers which parts of your site they may visit. It’s the first file well-behaved bots read, and it’s where you link your XML sitemap.
This generator builds a correct file for you: allow crawling by default, keep bots out of low-value areas like admin, cart and internal search, link your sitemap, and decide separately whether to block AI training crawlers such as GPTBot, ClaudeBot and Google-Extended, while AI search bots stay allowed.
How to Create Your robots.txt
- Step 1
Choose the default
Allow crawling for live sites. Block everything only on staging or private sites.
- Step 2
List paths to keep crawlers out of
Admin areas, carts, checkout, internal search results and endless filter URLs.
- Step 3
Add your sitemap and upload
Paste your sitemap URL, copy the file, and upload it to your site root. Test it with the robots.txt Validator.
Try the AI Crawl Intelligence
See which search and AI bots actually visit your site and what they can reach. Free to start, no credit card.
robots.txt Basics
Industry figures come from public studies and change often; check the original source before quoting them.
robots.txt Mistakes
Shipping “Disallow: /” from staging
Fix: This blocks Google from your entire live site. Check robots.txt after every launch.
Using robots.txt to hide pages from Google
Fix: Blocked pages can still be indexed from links. Use a noindex tag instead, and don’t block the page.
Blocking CSS and JavaScript
Fix: Google needs them to render your pages. Don’t block asset folders.
Treating it as security
Fix: robots.txt is public and only a request. Protect private areas with logins.
Tips
Keep it short
Separate AI training from AI search
Test before you upload
Questions about the Robots.txt Generator
Where does robots.txt go?
In the root of your domain, so it opens at https://yoursite.com/robots.txt. Each subdomain needs its own.
Will blocking GPTBot remove me from ChatGPT?
Blocking GPTBot stops OpenAI using your pages for training. ChatGPT search uses OAI-SearchBot and browsing uses ChatGPT-User, which this generator leaves allowed.
Does Google respect Crawl-delay?
No, Googlebot ignores Crawl-delay, so this generator leaves it out.
Can robots.txt remove a page from Google?
No. Use a noindex tag and let Google crawl the page so it can see the tag.
Is it free?
Yes, unlimited, and it runs in your browser.
See which bots actually visit you
AI Crawl Intelligence in the app shows which search and AI crawlers reach your site.