About the Robots.txt Generator
Build a valid robots.txt without memorizing the syntax. Choose which paths to block, add your sitemap, start from a WordPress or shop template, and set rules for search engines and AI crawlers individually.
Key features
- Allow or block the whole site, or specific paths
- Templates for WordPress, online shops and staging sites
- Per-crawler rules for Google, Bing, Apple, Yandex and more
- One-click blocking of AI training crawlers
- Sitemap and crawl-delay lines
- Copy or download a ready-to-upload robots.txt
Build a robots.txt file without the guesswork
A robots.txt file tells crawlers which parts of your site they may visit. A mistake can hide your whole site from search engines, so this generator produces valid syntax and explains each option.
How robots.txt works
User-agent: * ← which crawler the rules apply to
Disallow: /admin/ ← don't crawl URLs starting with /admin/
Allow: /admin/help/ ← exception to the rule above
User-agent: GPTBot
Disallow: / ← block this crawler from the whole site
Sitemap: https://example.com/sitemap.xml
- A crawler uses the most specific group that names it, falling back to
*. - Within a group, the longest matching rule wins;
Allowwins a tie. *matches any characters and$marks the end of the URL:Disallow: /*.pdf$.
AI crawlers
| User agent | Operator | What it does |
|---|---|---|
| GPTBot | OpenAI | Collects training data |
| OAI-SearchBot | OpenAI | Indexes for ChatGPT search |
| ClaudeBot | Anthropic | Collects training data |
| Claude-SearchBot | Anthropic | Indexes for Claude search |
| Google-Extended | Controls Gemini training use (not Search) | |
| Applebot-Extended | Apple | Controls Apple AI training use |
| CCBot | Common Crawl | Public dataset used by many AI models |
| PerplexityBot | Perplexity | Indexes for Perplexity answers |
"Block AI training" blocks the training crawlers but keeps AI search crawlers that can send visitors to your site. Crawlers are listed by their documented user-agent names; respecting robots.txt is voluntary for any bot.
Common mistakes
Disallow: /left over from staging, which blocks the entire site.- Blocking CSS and JavaScript, which stops Google from rendering pages properly.
- Using robots.txt to hide private pages. The file is public; protect private content with a login.
- Relative sitemap URLs. Sitemap lines must be absolute (https://…).
Related tools
Test your rules with the Robots.txt Tester, and create the sitemap to reference with the XML Sitemap Generator.
How to use the Robots.txt Generator
- Fill in the fields or choose a template.
- Watch the output and previews update as you type.
- Copy the generated code or download the file.
- Add it to your website, then verify it with the matching checker or tester.