robots.txt Examples & Templates
Copy proven robots.txt templates used by real websites.
Robots.txt examples help you understand how to control search engine crawling — and these copy-ready templates cover the most common configurations used by real websites, from simple blogs to large e-commerce stores. Whether you're a developer setting up a new site, a webmaster managing crawl budget, or a store owner protecting your checkout pages, there's a ready-made template here for you. Each example follows the Robots Exclusion Protocol and has been tested against Google's guidelines. Open any in our free robots.txt generator to customise it to your exact needs. For a deeper understanding of how these directives work, read our guide to robots.txt or review the best practices. Once you've copied a template, test it using our robots.txt tester to see which URLs are allowed or blocked. If you're using platforms like Shopify or Wix, see our dedicated robots.txt guides by use case.
Common robots.txt examples for different use cases
Basic Allow All (Standard)
The most common setup. Tells all search engines they are free to crawl the entire website.
Use this when you have a public site with no private sections and you want all pages indexed normally.
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xmlBlock All Crawlers (Dev / Staging)
Critical for staging environments or sites currently under construction to prevent accidental indexing.
Use this when your website is under development or on a staging URL that should never appear in search results.
User-agent: *
Disallow: /Block Specific Bad Bots
Stops known scrapers, AI crawlers, or aggressive SEO tools from eating up your server bandwidth.
Use this when you want to block specific AI training crawlers or SEO audit bots without restricting Google or Bing.
User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: GPTBot
Disallow: /E-Commerce (Block Cart & Checkout)
Essential for stores to prevent Google from indexing empty carts, user accounts, and checkout processes.
Use this for any online store — these pages waste crawl budget and can create thin-content issues if indexed.
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?add-to-cart=*
Sitemap: https://store.com/sitemap.xmlWordPress Optimised robots.txt
The recommended configuration for WordPress sites — blocks admin and login pages while keeping everything required for plugins to function.
Use this for any WordPress site. It protects your admin panel, reduces crawl waste, and includes a sitemap pointer for Google.
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-login.php
Disallow: /xmlrpc.php
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/sitemap.xmlNews Site (Google News Specific)
Treats standard Googlebot differently than the Googlebot-News crawler.
Use this if you run a news or media publication and want full Google News access while still restricting admin and filtered pages.
User-agent: Googlebot-News
Allow: /
User-agent: *
Disallow: /admin/
Disallow: /authors/*?sort=
Sitemap: https://news.com/news-sitemap.xmlCustomize any of these examples
Load any template into our visual builder to adjust rules, test URL paths, and validate syntax — all in one place.
Used by developers, SEO professionals, and website owners worldwide.
Robots.txt Guides by Use Case
Explore ready-to-use robots.txt configurations for popular platforms and use cases.
Frequently Asked Questions
Common questions about robots.txt configurations and how to use these templates.
What is the best robots.txt example to start with?
For most public websites, the Basic Allow All template is the right starting point — it opens your entire site to crawlers and adds a sitemap reference. If you're on WordPress, use the WordPress template from our generator, which blocks the admin panel and login page while keeping everything else accessible.
Should I block all crawlers with Disallow: /?
Only on development or staging environments. Blocking all crawlers on a live production site will prevent Google from indexing your pages, which means your site won't appear in search results. If your site is already live, never use Disallow: / under User-agent: * unless you intend to remove it entirely from search engines.
How do I create a robots.txt file for WordPress?
Use our WordPress template (available on the Generator page) which blocks /wp-admin/, /wp-includes/, and /wp-login.php — while keeping /wp-admin/admin-ajax.php open so plugins continue to function. Place the generated file at yoursite.com/robots.txt and include your XML sitemap URL at the bottom.
Can robots.txt block bad bots and scrapers?
It can block well-behaved bots that respect the protocol — including AI crawlers like GPTBot and most commercial SEO tools. However, malicious scrapers and spam bots generally ignore robots.txt entirely. For those, you'll need server-level rate limiting or a web application firewall rather than relying on this file.
Do I need a different robots.txt for each subdomain?
Yes. Each subdomain is treated as a separate site by search engines. A robots.txt at yoursite.com has no effect on blog.yoursite.com or shop.yoursite.com. Each subdomain needs its own robots.txt file placed at its own root.
How do I test robots.txt after copying a template?
Paste the template into our free robots.txt tester, click Validate & Test, then enter URL paths like /admin/ or /products/ to instantly see which are blocked or allowed. Always test before uploading to your live site — a single incorrect Disallow rule can block entire sections from Google.
How do I check if a URL is blocked by robots.txt?
Open our robots.txt tester, paste your file, and enter the URL path you want to verify (e.g. /checkout/). Select a user-agent like Googlebot and click Test. The tool shows whether that URL is Allowed or Blocked — and which specific rule is responsible.
Does robots.txt affect SEO?
Yes. Robots.txt controls which pages Googlebot can crawl. Blocking important pages stops them from being indexed and appearing in search results. Accidentally blocking your homepage or product pages is one of the most damaging technical SEO mistakes. Always use our tester to verify rules before publishing.
What is the correct robots.txt format?
Each rule block starts with 'User-agent:' followed by 'Disallow:' and/or 'Allow:' directives, one per line. End the file with a 'Sitemap:' line pointing to your XML sitemap. The file must be plain text, UTF-8 encoded, and placed at the root of your domain (e.g. yoursite.com/robots.txt).