Robots.txt for React (Example, Template & Best Practices)
QUICK SUMMARY
- Place
robots.txtin the/public/directory — build tools copy it to the output folder automatically - React SPAs render client-side, which creates inherent SEO limitations robots.txt cannot fix
- Block admin, dashboard, and API paths to protect non-indexable routes
- Generate a sitemap of your React Router routes and reference it in
Sitemap: - For SEO-critical content, consider migrating to Next.js or Remix for server-side rendering
USEFUL TOOLS
A robots.txt for a React app works like any other — it tells crawlers which paths to avoid — but React's single-page application (SPA) architecture introduces unique SEO considerations. A plain React app built with Create React App or Vite renders entirely in the browser via JavaScript. Googlebot can execute that JavaScript, but server-rendered HTML is always more reliable for indexing. robots.txt controls access; it won't improve how well Google can render your SPA.
React robots.txt template
This template is suitable for most React SPAs deployed on Netlify, Vercel, or a static host. Adjust paths to match your app's routing structure and update the Sitemap URL to your production domain:
User-agent: *
Disallow: /admin/
Disallow: /dashboard/
Disallow: /api/
Sitemap: https://yourdomain.com/sitemap.xmlWhat each rule does
- /admin/: Admin interfaces in React apps are typically protected by client-side authentication. Blocking them in robots.txt prevents crawlers from attempting to index them, though server-side protection is always required for real security.
- /dashboard/: Authenticated user dashboards with no public indexable content. Blocking these keeps personalised, session-based pages out of Google's index.
- /api/: If your React app communicates with an API on the same domain, block these endpoints — they return JSON responses, not HTML pages, and have no value in search results.
Once deployed, test this robots.txt using our robots.txt tester to confirm your public-facing routes are accessible and no important pages are blocked. For more context on how robots.txt affects React apps, read our robots.txt best practices guide.
Common React robots.txt mistakes
These errors are easy to make in React projects and each one has a real impact on crawling.
Placing robots.txt in /src/ instead of /public/
React build tools (CRA, Vite) only copy static files from /public/ to the build output. A robots.txt in /src/ will never be served by your web server.
Using robots.txt as a security measure
React Router guards, protected routes, and authentication logic are client-side only. robots.txt adds a crawl signal but does not protect routes — always use server-side authentication for sensitive pages.
Forgetting to update the Sitemap URL for production
The Sitemap: directive in your robots.txt must use an absolute URL matching your production domain. If you copy from staging or localhost, crawlers will follow a broken sitemap link.
Not generating a sitemap for SPA routes
React Router routes don't automatically produce a sitemap. Use a tool like react-router-sitemap, vite-plugin-sitemap, or a custom build script to generate sitemap.xml and reference it in your robots.txt.
React robots.txt best practices
Follow these principles to get the best SEO results from your React project's robots.txt setup.
Place robots.txt in /public/, not /src/
This is the only directory that React build tools copy into your final build output. Files in /src/ are compiled and bundled, not served as-is.
Generate a sitemap of your React Router routes
Even without SSR, a sitemap helps Google discover and prioritise your client-side routes. Use a build plugin or script to generate sitemap.xml at build time.
Consider Next.js or Remix for SEO-critical content
If your app has content pages that must rank in search results, server-side rendering provides far more reliable indexing than a pure SPA. robots.txt won't fix a rendering gap.
Test after each production deployment
Use our free robots.txt tester to verify the file is being served correctly and no important routes are accidentally blocked.
When NOT to use these rules
Applying these directives in the wrong context can harm your app's visibility. Keep these limits in mind:
- Do NOT expect robots.txt to fix React SPA indexing issues — only server-side rendering or static generation addresses the root cause.
- Do NOT block /static/ or your build output asset directories — Googlebot may need CSS and JavaScript files to render your pages.
- Do NOT use Disallow: / to hide a development build — use environment-specific deployment configurations or HTTP authentication instead.
Configure robots.txt for your React app
Test your existing file for errors, or build the correct configuration from scratch — free, no account required.
Related robots.txt guides
More platform-specific guides in the series.
Frequently Asked Questions
Common questions about robots.txt in React projects.
Where do I put robots.txt in a React app?
For Create React App (CRA) or Vite-based React projects, place robots.txt in the /public/ directory. When you run npm run build, the build tool copies everything from /public/ into the output folder (typically /build/ for CRA or /dist/ for Vite), making robots.txt available at yoursite.com/robots.txt. Never place it in /src/ — it won't be included in the build output.
Can Google crawl a React SPA?
Yes, but with limitations. Googlebot can execute JavaScript to render client-side React apps, but it takes longer than crawling server-rendered HTML. If your content depends entirely on JavaScript to appear (for example, data fetched via useEffect after page load), Google may not index it reliably. For content-critical pages that need to rank in search, consider migrating to Next.js, Remix, or using a pre-rendering service.
How does robots.txt work with React Router?
React Router handles routing client-side via the browser's History API. From a server perspective, all URLs resolve to the same index.html file. This means robots.txt Disallow rules still apply at the URL level — Googlebot won't attempt to crawl disallowed paths — but from a rendering standpoint, those routes are controlled by your JavaScript, not separate server files. Block admin and sensitive routes in robots.txt as an additional signal even if authentication guards them.
Should I use robots.txt to improve React SEO?
Robots.txt controls access, not rendering quality. It won't fix the core SPA indexing challenge (which requires SSR or SSG). However, a well-configured robots.txt does help by: (1) directing crawl budget away from admin and API routes, (2) providing a Sitemap: directive that helps Google discover all your client-side routes, and (3) preventing private routes from appearing in search results.
What is the difference between robots.txt in React vs Next.js?
In a plain React SPA, robots.txt is a static file in /public/ with no framework-specific conventions. In Next.js, you have two options: a static file in /public/robots.txt or a dynamic app/robots.ts file in the App Router. Next.js also has better SEO foundations overall because it supports server-side rendering and static generation — meaning Googlebot reads HTML directly rather than waiting for JavaScript to execute.
For more configuration examples, see our robots.txt examples and best practices guide.