Robots.txt for Blogs — Example, Template & Best Practices
QUICK SUMMARY
- A robots.txt file for a blog controls which pages search engines can crawl and index
- Block internal search results and admin pages — not category or post pages
- Never block CSS, JavaScript, or images on your blog
- Tag pagination and date archives are often worth blocking on content-heavy sites
- Always include a
Sitemap:directive pointing to your post sitemap
USEFUL TOOLS
A robots.txt file for a blog helps control how search engines crawl your site — keeping Googlebot focused on your best content and away from low-value pages like internal search results, admin panels, and paginated archives. The goal is precision: block only what genuinely has no SEO value, and leave everything else open.
Blog robots.txt template
This template is suitable for most WordPress and other CMS-based blogs. Adjust the paths for your specific platform, or use our robots.txt generator to build a customised version:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search
Disallow: /tag/*/page/
Disallow: /date/
Sitemap: https://yourblog.com/sitemap.xmlWhat each rule does
- /wp-admin/ with Allow for admin-ajax.php: Blocks the WordPress admin panel while keeping plugin functionality working. Always include both lines together.
- /?s= and /search: Internal blog search results are thin, auto-generated content. Blocking them prevents duplicate content issues and crawl budget waste.
- /tag/*/page/: The first page of a tag archive may have SEO value. Deeper paginated pages (page 2, 3, etc.) typically don't — this rule blocks only those.
- /date/: Date-based archive pages (e.g. /2023/01/) are often near-duplicates of your category or homepage content. Block them if they don't serve a specific SEO purpose on your blog.
Once you've set up your rules, test this robots.txt using our robots.txt tester to confirm your posts, pages, and category pages are all accessible. For more templates, browse the robots.txt examples library.
Common blog robots.txt mistakes
These errors can silently reduce your blog's search visibility.
Blocking CSS or JavaScript files
Google needs your theme's CSS and JS to render pages and evaluate Core Web Vitals. Blocking these files can cause rendering failures and unexpected ranking drops.
Over-blocking archive pages
Category pages, monthly archives, and tag pages can rank well if they contain unique, focused content. Only block archive pages that are genuinely thin or duplicated elsewhere.
Missing the Sitemap directive
Your blog's XML sitemap tells crawlers where to find every post. Without a Sitemap: line in robots.txt, Google relies on link discovery alone, which can miss new content for days.
Using incorrect path prefixes
Robots.txt paths are prefix-matched. Disallow: /blog blocks /blog, /blog/, /blog-post, and /blogger-tips. Always be precise with trailing slashes and test every rule with a URL path tester.
Blog robots.txt best practices
Follow these principles to keep your blog's crawl configuration effective.
Block internal search, not category pages
Search result pages have no standalone ranking value. Category and tag pages often do. Be selective — only block what adds no SEO value on your specific blog.
Include your post sitemap URL
Add Sitemap: https://yourblog.com/sitemap.xml at the bottom. On WordPress, use the Yoast or Rank Math sitemap URL so Google finds all your posts automatically.
Never block images or media
Images drive Google Image Search traffic and support rich snippets. Never add Disallow rules that cover your media library or image directories.
Test after every change
Use our robots.txt tester to confirm your most important posts are still accessible to Googlebot before publishing any update to your file.
When NOT to use these rules
Applying these directives in the wrong context can harm your blog's SEO. Keep these limits in mind:
- Do NOT block CSS or JavaScript files — Google needs them to render your pages correctly and evaluate Core Web Vitals scores.
- Do NOT block all category or tag pages — well-structured category pages often rank well and drive consistent long-term traffic.
- Do NOT use Disallow: /wp-content/ — it blocks all your images, themes, and plugin assets, preventing Google from rendering your pages.
Get your blog's robots.txt right
Test your existing file instantly or generate a new one — free, no account needed.
Related robots.txt guides
More platform-specific guides in the series.
Frequently Asked Questions
Common questions about robots.txt configuration for blogs.
What should I block in my blog's robots.txt?
For most blogs, the highest-value blocks are: internal search result pages (/?s= or /search), admin areas (/wp-admin/ for WordPress), tag pagination beyond the first page (/tag/*/page/), and date-based archive pages (/date/) if they contain the same content as category or post pages. Block only what genuinely has no SEO value on your specific blog.
Should I block /tag/ pages on my blog?
It depends. Tag pages can rank well if each tag has a focused set of posts and unique introductory content. If your tags are broad, overlapping, or contain only 1–2 posts each, consider blocking them or using noindex meta tags instead. Noindex is often preferred over robots.txt blocking because it allows crawling without indexing.
Should I block author pages?
Only if your blog has a single author. On a single-author blog, /author/yourname/ pages are near-duplicates of your category pages and add no unique value. On multi-author blogs, author pages can rank for author-specific queries and build topical authority — leave them open in that case.
How do I prevent internal search results from being indexed?
Add Disallow: /?s= (WordPress) or Disallow: /search (other platforms) to your robots.txt. This prevents Googlebot from crawling search result pages, which are typically thin, session-specific content with no unique SEO value. You can also add a noindex meta tag to search result templates for belt-and-suspenders coverage.
For more SEO configuration guidance, see our robots.txt examples and best practices guide.