How to use the Robots.txt Generator
1. Start from a preset or build your own rules from scratch.
2. Choose a bot from the dropdown (or type a custom one) and set which paths to Allow or Disallow.
3. Add your sitemap URL, then copy or download the file and upload it to your website root as robots.txt.
What is it?
The robots.txt file lives at the root of your website (e.g., https://example.com/robots.txt) and tells search engine crawlers which pages they are allowed or not allowed to visit. It's the first file bots check before crawling your site. Misconfiguring it can accidentally block your entire site from search engines.
Common use cases: blocking /admin/, /api/, /checkout/, staging environments, and internal search result pages. The Sitemap directive helps crawlers discover all your pages efficiently. You can add separate blocks for different bots — search engines respect their matching block plus the wildcard (*) block as a fallback. AI bots like GPTBot (OpenAI), CCBot (Common Crawl), and anthropic-ai can be selectively blocked if you don't want your content used for AI training. Note: robots.txt is advisory — malicious bots may ignore it. Never use it to hide sensitive content.
Example
User-agent: * Allow: / User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: anthropic-ai Disallow: /
User-agent: * Allow: / User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: anthropic-ai Disallow: / Sitemap: https://example.com/sitemap.xml
GPTBot is OpenAI's crawler. Many site owners block it to prevent their content being used for AI training.
Benefits
Prevent sensitive pages from appearing in search results
Correctly configured Disallow directives keep admin panels, staging paths, and private pages out of Google's index.
Allow trusted bots while blocking scrapers and crawlers
Grant full access to Googlebot and Bingbot while blocking scrapers, competitive intelligence tools, and unwanted crawlers.
Link your sitemap directly in robots.txt for faster discovery
The Sitemap: directive tells crawlers exactly where to find your XML sitemap — speeding up discovery of new and updated pages.
Prevent accidental de-indexing with validated directives
An incorrect Disallow: / removes your entire site from search results — validate before deploying to avoid a catastrophic mistake.
Frequently asked questions
How do I allow all crawlers to access everything?
The simplest valid robots.txt is just: User-agent: * with no Disallow directive. Use the 'Allow All' preset to generate this instantly.
What is the difference between robots.txt and noindex?
robots.txt prevents crawlers from visiting the URL entirely. A noindex meta tag allows crawling but tells search engines not to index the page. If you want to stop a page appearing in search results, use noindex — not robots.txt. Blocked pages can still appear in search results via external links.
Do different crawlers need separate rule blocks?
Yes — you can create separate User-agent blocks for different bots with different rules. Crawlers read only their matching block plus the wildcard (*) block as a fallback. Order within a block matters; order of blocks does not.
Which AI bots should I block?
The most common AI training crawlers are GPTBot (OpenAI), ChatGPT-User (OpenAI), CCBot (Common Crawl / used by many LLMs), anthropic-ai (Anthropic/Claude), Google-Extended (Google Gemini training), and PerplexityBot. Use the 'Block AI Bots' preset to block all of them at once.
What is Crawl-delay?
Crawl-delay tells bots how many seconds to wait between requests. This reduces server load from aggressive crawlers. Note: Googlebot ignores Crawl-delay — use Google Search Console's crawl rate settings instead.