Robots.txt for WordPress
QUICK SUMMARY
- A robots.txt file in WordPress controls how search engines crawl your site
- Place the file at
yoursite.com/robots.txt - Always block
/wp-admin/but allow/wp-admin/admin-ajax.php - Never block
/wp-content/— it contains your CSS, JS, and images - Always include a
Sitemap:directive at the bottom
A robots.txt file in WordPress helps control how search engines crawl your site — and the default configuration only covers the bare minimum. A properly optimised WordPress robots.txt protects your admin panel, prevents low-quality pages from being crawled, and helps Google focus on your content that actually ranks. This guide walks you through everything — from the default file to the optimised version, common mistakes, and how to test your setup.
The default WordPress robots.txt
If you don't have a physical robots.txt file at your root, WordPress generates a virtual one automatically. It looks like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpThis is a reasonable starting point, but it leaves internal search result pages, plugin directories, login pages, and XML-RPC endpoints open to crawlers — all of which waste your crawl budget and add no SEO value.
Default vs. optimised: what's the difference?
| Rule | Default WP | Optimised |
| Disallow: /wp-admin/ | ✓ | ✓ |
| Allow: /wp-admin/admin-ajax.php | ✓ | ✓ |
| Disallow: /wp-includes/ | ✗ | ✓ |
| Disallow: /wp-login.php | ✗ | ✓ |
| Disallow: /xmlrpc.php | ✗ | ✓ |
| Disallow: /?s= (search results) | ✗ | ✓ |
| Sitemap: directive | ✗ | ✓ |
The recommended WordPress robots.txt template
For most WordPress sites aiming for strong technical SEO, this is the template we recommend. You can generate and customise it using our free robots.txt generator:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-includes/
Disallow: /wp-login.php
Disallow: /xmlrpc.php
Disallow: /?s=
Disallow: /search/
Sitemap: https://yourdomain.com/sitemap_index.xmlWhy block these specific paths?
- /wp-admin/ and /wp-login.php: Your admin login screen and dashboard should never appear in Google. Blocking them is non-negotiable.
- /wp-includes/: Contains WordPress core files. Crawlers don't need to index this directory — it's only useful to your server.
- /xmlrpc.php: An older WordPress API endpoint frequently targeted by bots. Blocking it reduces server load from automated attacks.
- /?s= and /search/: Internal WordPress search result pages are low-quality, near-duplicate content. Letting Google index them wastes crawl budget and can trigger thin-content penalties.
Common WordPress robots.txt mistakes
These are the errors we see most often — each one can silently hurt your SEO.
Blocking /wp-content/
This blocks your theme files, images, and plugin assets. Google can't render your pages without them, which causes ranking drops and Search Console errors.
Blocking CSS and JavaScript files
Google needs to fetch and render your CSS/JS to evaluate page experience signals like Core Web Vitals. Blocking these files hides your design from Google's crawler.
Forgetting the Sitemap directive
Without a Sitemap: line, crawlers won't automatically find your XML sitemap. Always add Sitemap: https://yoursite.com/sitemap_index.xml at the bottom of your file.
Over-blocking important content pages
Some guides suggest blocking /category/, /tag/, or /author/ archives. Only do this if those pages have no SEO value on your specific site — on many sites, category pages rank well.
Forgetting Allow: /wp-admin/admin-ajax.php
After adding Disallow: /wp-admin/, many people forget to re-allow admin-ajax.php. Without it, plugin-powered forms, search boxes, and dynamic elements stop working for users.
WordPress robots.txt best practices
Follow these principles and your file will be in better shape than most WordPress sites on the web.
Always include a Sitemap directive
Add Sitemap: https://yoursite.com/sitemap_index.xml at the very end of the file. Every major crawler reads this and uses it to prioritise your pages.
Keep rules simple and minimal
Only add Disallow rules for pages that genuinely have no SEO value. Every unnecessary rule increases the chance of accidentally blocking something important.
Never block CSS, JS, or images
These assets are essential for Google to render your pages. Blocking them is a common cause of poor Core Web Vitals scores and unexpected ranking drops.
Test after every change
Use Google Search Console's robots.txt Tester or our Generator's built-in URL tester to verify your rules before publishing them to your live site.
How to edit your WordPress robots.txt
There are three ways to update this file in WordPress:
Method 1: Using an SEO plugin (recommended)
If you use Yoast SEO, Rank Math, or All in One SEO, all three provide a UI to edit the file from your WordPress dashboard — no FTP or server access needed.
- Yoast SEO: Go to SEO → Tools → File Editor.
- Rank Math: Go to Rank Math → General Settings → Edit robots.txt.
- All in One SEO: Go to All in One SEO → Tools → Robots.txt Editor.
Method 2: Upload via FTP or cPanel
Use our generator to build and download the file, then upload it to your public_html folder using FileZilla or your hosting control panel. Once a physical file is in place, WordPress stops generating the virtual one.
Method 3: Edit via hosting file manager
Most WordPress hosts (SiteGround, Bluehost, WP Engine) provide a file manager in their dashboard. Open public_html/robots.txt and paste your content directly.
A note on /wp-content/ — do not block it
Blocking /wp-content/ is one of the most destructive WordPress SEO mistakes. Your theme CSS, JavaScript, and plugin assets all live there. Google must fetch those files to render and evaluate your pages. Blocking this folder causes "Page cannot be reached" errors in Search Console and prevents Google from assessing your Core Web Vitals. Leave it open — always. For more on this topic, read the robots.txt best practices guide.
If you'd like to compare different configuration approaches, our robots.txt examples page has ready-to-copy templates for WordPress, e-commerce, news sites, and more. Once you've updated your file, always validate robots.txt using our free checker to confirm every rule is working as expected.
Create your WordPress robots.txt now
Use our free generator to build, preview, and download the perfect file for your WordPress site — no account required, everything runs in your browser.
Used by developers, SEO professionals, and website owners worldwide.
Related robots.txt guides
More platform-specific guides in the series.
Frequently Asked Questions
Common questions about setting up robots.txt on a WordPress website.
Where is robots.txt in WordPress?
If a physical file exists, it's in the root directory of your WordPress installation — the same folder as wp-config.php and the wp-content folder. You can access it via FTP, your hosting file manager, or an SEO plugin. If no physical file exists, WordPress automatically generates a virtual robots.txt at yoursite.com/robots.txt using minimal default rules.
How do I edit robots.txt in WordPress?
There are three ways: (1) Use an SEO plugin — Yoast SEO (SEO → Tools → File Editor), Rank Math (General Settings → Edit robots.txt), or All in One SEO (Tools → Robots.txt Editor). (2) Edit the file directly via FTP or your hosting file manager at /public_html/robots.txt. (3) Use our robots.txt generator to build and download the file, then upload it to your root directory.
What is the best robots.txt for WordPress?
The best WordPress robots.txt blocks /wp-admin/, /wp-login.php, /wp-includes/, /xmlrpc.php, and internal search pages (/?s=). It should explicitly allow /wp-admin/admin-ajax.php so plugins continue to work, and always include a Sitemap: directive pointing to your XML sitemap URL.
Should I block /wp-admin/ in robots.txt?
Yes — always. Your WordPress admin panel should never appear in search results. Block /wp-admin/ with Disallow: /wp-admin/ but always add Allow: /wp-admin/admin-ajax.php on the next line. Without that Allow rule, many plugins (contact forms, live search, dynamic content) will break because they rely on admin-ajax.php.
Should I block /wp-content/ in robots.txt?
No. This is one of the most common WordPress SEO mistakes. Your theme CSS, JavaScript, and images all live inside /wp-content/. If you block it, Google cannot render your pages correctly, which can hurt your rankings and trigger 'Page cannot be reached' errors in Search Console. Leave /wp-content/ fully open.
Does WordPress generate robots.txt automatically?
Yes — if you don't have a physical robots.txt file at your root, WordPress creates a virtual one. It only contains two lines: Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php. This minimal file misses a lot of important optimisations. For best results, create a custom file using an SEO plugin or our generator tool.
How do I test my WordPress robots.txt?
Visit yoursite.com/robots.txt in a browser to confirm it's accessible. For a deeper test, use Google Search Console's robots.txt Tester (under Settings) — it shows you exactly how Googlebot reads your file and highlights any parsing errors. You can also use our built-in URL tester on the Generator page.