Robots.txt for eCommerce — Example, Template & SEO Guide
QUICK SUMMARY
- A robots.txt file for eCommerce protects crawl budget and keeps product pages prioritised
- Always block
/cart/,/checkout/, and/account/ - Block faceted navigation parameters to prevent duplicate collection pages
- Never block product images, CSS, or JavaScript
- Always include a
Sitemap:directive pointing to your product sitemap
USEFUL TOOLS
A robots.txt file for eCommerce helps control how search engines crawl your store — directing Googlebot away from cart pages, checkout flows, and duplicate filter URLs, and towards the product and category pages that actually drive revenue. For large stores with thousands of SKUs, a well-configured file can make a significant difference to how quickly Google discovers and indexes new products.
eCommerce robots.txt template
This template works across most eCommerce platforms. Adapt the parameter names to match your specific URL structure, then use our robots.txt generator to customise and download it:
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /orders/
Disallow: /wishlist/
Disallow: /*?sort=
Disallow: /*?page=
Disallow: /search/
Sitemap: https://yourstore.com/sitemap.xmlWhat each rule does
- /cart/ and /checkout/: Transactional pages with no indexable value. Crawling them wastes budget that should be spent on products.
- /account/ and /orders/: Customer-specific pages with personalised content. Not indexable and not relevant to search.
- /wishlist/: User-generated, session-specific pages. Block to prevent temporary and duplicate content from entering Google's index.
- /*?sort= and /*?page=: Pagination and sort parameters generate thousands of near-duplicate collection pages on large stores. These are one of the biggest crawl budget drains in eCommerce.
- /search/: Internal search result pages produce thin content. Blocking them prevents duplicate content and wasted crawl budget.
Once your file is configured, test this robots.txt using our robots.txt tester to confirm all your product and category URLs are allowed. For platform-specific examples, browse our robots.txt examples library.
Common eCommerce robots.txt mistakes
These are the errors most likely to harm a store's search visibility.
Blocking category or product pages
A broad Disallow rule that catches /products/ or /categories/ removes your most valuable pages from Google's index. Always check with a URL path tester before going live.
Ignoring faceted navigation
Unblocked filter and sort parameters create thousands of near-duplicate pages. On large catalogues, this is one of the most common causes of crawl budget waste and thin-content penalties.
Missing the Sitemap directive
For eCommerce sites with large product catalogues, the sitemap is essential for ensuring Googlebot discovers every product. Always add Sitemap: at the bottom of your robots.txt.
Blocking image files or CDN paths
Product images are a significant source of Google Image Search traffic for eCommerce. Never block image directories or CDN paths — doing so prevents Google from indexing your product visuals.
eCommerce robots.txt best practices
Follow these principles to protect your crawl budget and maximise product discoverability.
Block all transactional pages
Cart, checkout, order confirmation, and account pages should all be blocked. They generate no organic search value and waste crawl budget.
Block faceted URL parameters systematically
Map out every query parameter your store uses for filters and sorting. Add Disallow rules for each one to prevent duplicate collection pages.
Submit your product sitemap to Google
Large stores benefit from a dedicated product sitemap. Include the Sitemap: directive in robots.txt and submit it through Google Search Console to speed up product discovery.
Test every rule against real product URLs
Use our robots.txt tester to verify that your most important product and category pages are accessible to Googlebot before deploying any changes.
When NOT to use these rules
Applying these directives in the wrong context can harm your store's SEO. Keep these limits in mind:
- Do NOT block /products/ or /categories/ — these are your highest-value pages. Blocking them removes your entire product catalogue from Google's index.
- Do NOT over-block filter parameters — only target query strings that create duplicate pages without unique ranking value, not all query parameters.
- Do NOT block image directories or CDN paths — product images drive Google Image Search traffic and support visual rich results.
Optimise your eCommerce robots.txt
Test your existing file or build a new one tailored to your store — free, no login required.
Related robots.txt guides
More platform-specific guides in the series.
Frequently Asked Questions
Common questions about robots.txt configuration for online stores.
Should I block /cart and /checkout in robots.txt?
Yes. Cart and checkout pages have no standalone SEO value — they are transactional steps that users reach through your product and category pages. Blocking them prevents crawlers from wasting crawl budget on pages that will never rank, and keeps Googlebot focused on your product and collection pages.
How do I stop faceted navigation from creating duplicate pages?
Use Disallow rules for the URL parameters that generate duplicate or near-duplicate pages — for example, Disallow: /*?sort= and Disallow: /*?page=. The exact parameter names depend on your eCommerce platform. Blocking these is one of the highest-impact technical SEO changes for large product catalogues.
Should I block account and order pages?
Yes. Customer account pages, order history, wishlists, and address books are personalised and non-indexable content. Blocking /account/, /orders/, and /wishlist/ keeps crawlers out of content they cannot meaningfully index.
What is crawl budget and why does it matter for eCommerce?
Crawl budget is the number of pages Googlebot will crawl on your site within a given time period. Large eCommerce sites with thousands of products can easily exhaust their crawl budget on low-value pages (cart, checkout, filters, search results) before Googlebot reaches all their important product pages. A well-configured robots.txt directs that budget to the pages that actually drive sales.
For more examples and technical guidance, see our robots.txt examples and best practices guide.