Robots.txt for Next.js (Example, Template & Best Practices)
QUICK SUMMARY
- Place
robots.txtin your Next.js/public/directory to serve it at the root - Next.js 13.3+ App Router also supports
app/robots.tsfor dynamic generation - Block
/_next/to prevent crawling of compiled build artifacts - Block
/api/to prevent indexing of JSON API routes - Use
next-sitemaporapp/sitemap.tsand reference the output in yourSitemap:directive
USEFUL TOOLS
Next.js is one of the most SEO-friendly React frameworks available — its server-side rendering and static generation mean Googlebot can read your pages without executing JavaScript. Your robots.txt for Next.js lives in the /public/ directory, and with Next.js 13.3+ you can even generate it dynamically via app/robots.ts. This guide walks through the recommended template, what each rule does, and how to test your configuration before deploying.
Next.js robots.txt template
Use this as a starting point for most Next.js projects. Adjust /admin/ to match your actual admin path, and customise the Sitemap URL. You can also use our robots.txt generator to build a version specific to your project:
User-agent: *
Disallow: /_next/
Disallow: /api/
Disallow: /admin/
Sitemap: https://yourdomain.com/sitemap.xmlWhat each rule does
- /_next/: The Next.js build output directory containing compiled JavaScript chunks, CSS modules, and static assets. These are build artifacts, not indexable pages — blocking them keeps crawl budget focused on your actual content.
- /api/: Next.js API routes (both Pages Router and App Router Route Handlers) return JSON or structured data, not HTML. Blocking them prevents crawlers from wasting resources on non-page endpoints.
- /admin/: Any admin interface specific to your application. Replace this with your actual admin path if different.
Once your file is in place, test this robots.txt using our robots.txt tester to verify your pages are accessible and no important routes are accidentally blocked. For more templates, see our robots.txt examples library.
Common Next.js robots.txt mistakes
These errors catch even experienced Next.js developers off guard.
Placing robots.txt outside /public/
Next.js only serves static files from the /public/ directory. If you place robots.txt in the project root or /src/, it will not be accessible at yoursite.com/robots.txt.
Forgetting robots.txt in static exports
When using next export, the output is in /out/. Ensure your robots.txt in /public/ is correct before building — it gets copied automatically, but the Sitemap URL must point to your production domain.
Blocking /_next/static/ without considering rendering mode
If your app uses client-side rendering (no getServerSideProps or static generation), Googlebot needs the JS bundles to render your pages. Blocking /_next/ in a CSR app can cause Google to see blank pages.
Missing the Sitemap directive
Next.js doesn't auto-generate a sitemap. You must either use next-sitemap, app/sitemap.ts (Next.js 13.3+), or a custom API route — and reference the output URL in your robots.txt.
Next.js robots.txt best practices
Follow these steps to keep your Next.js crawl configuration correct across every deployment.
Use /public/robots.txt for simplicity
A static file in /public/ works on every Next.js version, including Pages Router and App Router. No plugin required — just place the file and deploy.
Use app/robots.ts for environment-aware rules
Next.js 13.3+ lets you export a robots.ts file in your app/ directory. Use process.env to serve Disallow: / on staging and full open rules on production automatically.
Pair with next-sitemap for automatic sitemap generation
Install next-sitemap, configure it in next-sitemap.config.js, and it generates sitemap.xml at build time. Reference the output URL in your robots.txt Sitemap: directive.
Test after every deployment
Use our free robots.txt tester to confirm /_next/ is correctly blocked and your product pages are accessible to Googlebot after each deployment.
When NOT to use these rules
These directives can cause unintended consequences in certain Next.js configurations. Keep these limits in mind:
- Do NOT block /_next/static/ if your pages depend on client-side rendering — Googlebot needs the JS bundles to render CSR content.
- Do NOT block /api/ routes that serve public documentation or data used to build server-rendered pages.
- Do NOT ship your staging robots.txt (Disallow: /) to production — use app/robots.ts with environment variables to swap rules automatically.
Set up robots.txt for your Next.js project
Test your existing configuration or generate a new one from scratch — free, no account required.
Related robots.txt guides
More platform-specific guides in the series.
Frequently Asked Questions
Common questions about robots.txt in Next.js projects.
Where do I put robots.txt in Next.js?
Place robots.txt in the /public/ directory of your Next.js project. When you run next build, Next.js copies everything in /public/ to the root of your output, making robots.txt available at yoursite.com/robots.txt. In Next.js 13.3+ with the App Router, you can also generate robots.txt dynamically using app/robots.ts — but a static file in /public/ works for all Next.js versions.
Should I block /_next/ in robots.txt?
The /_next/ directory contains compiled JavaScript bundles, CSS chunks, and build artifacts used for client-side navigation. These are not indexable web pages, so blocking them prevents crawl budget waste. However, if your site relies heavily on client-side rendering (instead of SSR or SSG), Googlebot may need access to the JS bundles to render your content — in that case, leave /_next/ open and ensure your pages are server-rendered.
How do I add a sitemap in Next.js?
For Next.js 13.3+ App Router, add app/sitemap.ts to generate a dynamic sitemap automatically. For older versions or Pages Router, use the next-sitemap npm package — it generates sitemap.xml at build time and lets you reference it in robots.txt. Always add Sitemap: https://yourdomain.com/sitemap.xml to the bottom of your robots.txt file.
Can I use robots.ts instead of robots.txt in Next.js?
Yes — Next.js 13.3+ App Router supports a special app/robots.ts file that generates robots.txt programmatically. This is useful if your rules need to change based on environment variables (e.g., blocking all crawlers on staging, allowing all on production). Export a default function returning a MetadataRoute.Robots object. Both approaches produce the same result; static public/robots.txt is simpler and works on all Next.js versions.
How do I block Next.js API routes from being indexed?
Add Disallow: /api/ to your robots.txt. Next.js API routes return JSON responses, not HTML pages, so they have no SEO value and should not appear in search results. If you have specific public API documentation pages (not endpoints), you can override with Allow: /api/docs/ before the broader Disallow rule.
For more configuration examples, see our robots.txt examples and best practices guide.