Robots.txt for Static Sites (Example, Template & Best Practices)
QUICK SUMMARY
- Static sites (Netlify, Vercel, GitHub Pages) are the most crawler-friendly setup — all pages are pre-rendered HTML
- Place
robots.txtin your site's root output directory (e.g./public/for Gatsby/Astro) - Very few Disallow rules are needed — only block admin areas and platform-specific utility paths
- Always include a
Sitemap:directive pointing to your generated sitemap - Deploy robots.txt as part of your build pipeline, not independently
USEFUL TOOLS
A robots.txt for a static site is the simplest configuration of all. Because every page is pre-rendered HTML served directly from a CDN — on platforms like Netlify, Vercel, Cloudflare Pages, or GitHub Pages — there are almost no paths that need to be blocked. The file's primary purpose is to declare your sitemap location and suppress any platform-specific utility endpoints that shouldn't appear in search results.
Static site robots.txt template
This template covers the most common static site hosting patterns. Replace .netlify with your platform's equivalent if you're not on Netlify, and update the Sitemap URL to match your production domain. Build a customised version using our robots.txt generator:
User-agent: *
Disallow: /admin/
Disallow: /.netlify/
Disallow: /api/
Sitemap: https://yourdomain.com/sitemap.xmlWhat each rule does
- /admin/: If your static site uses a headless CMS with an admin interface (Netlify CMS, Decap CMS, Contentful preview), block the admin path from indexing.
- /.netlify/: Netlify's internal path for serverless functions and edge middleware. These endpoints return JSON or function responses — not web pages — so they should not be indexed. If you're on Vercel, this path doesn't apply; block
/api/instead. - /api/: Netlify Functions, Vercel API routes, or any serverless backend path on the same domain. JSON-returning endpoints have no SEO value.
Once deployed, test this robots.txt using our robots.txt tester to confirm all your pre-rendered pages are accessible. For more templates covering other platforms, see our robots.txt examples library.
Common static site robots.txt mistakes
Static sites need very few rules — so when mistakes happen, they're usually caused by adding too much, not too little.
Adding unnecessary Disallow rules on static sites
Static sites are almost entirely composed of pre-rendered HTML pages that should be indexed. Overly broad Disallow rules can accidentally remove good content from Google's index. Keep rules minimal — only block what genuinely has no SEO value.
Forgetting the Sitemap directive
Static site generators typically produce comprehensive XML sitemaps. If you don't reference your sitemap in robots.txt, crawlers must discover your pages through links alone — slowing down indexing of new content.
Stale robots.txt due to CDN caching
Static sites on Netlify and Vercel are served via CDN. If you update your robots.txt, the old version may be cached for hours or days. After changing your file, trigger a full redeploy — don't just upload the file.
Not accounting for Netlify Functions or Vercel API routes
Even 'static' sites often use serverless functions for forms, search, or APIs. Block these paths (/.netlify/ or /api/) in robots.txt so crawlers don't waste budget on JSON-returning endpoints.
Static site robots.txt best practices
Simple principles for static site crawl configuration that work across all hosting platforms.
Keep the robots.txt file as minimal as possible
Static sites are inherently crawler-friendly. The shorter your Disallow list, the less chance of accidentally blocking a good page. Add rules only when you have a specific, intentional reason.
Always include your sitemap URL
Static site generators create comprehensive sitemaps. Reference yours in robots.txt so Google can discover every pre-rendered page quickly — especially important after adding new content.
Deploy robots.txt as part of your build pipeline
Never update robots.txt independently from a full site deployment. Treating it as a build artifact — generated and deployed together with your pages — prevents version mismatches and CDN cache issues.
Test immediately after every deployment
Use our free robots.txt tester to verify your file is live, accessible, and your key pages are still open to crawlers after each deploy.
When NOT to use these rules
Applying unnecessary Disallow rules on static sites can hide content that should be indexed. Keep these limits in mind:
- Do NOT block HTML pages on static sites — they are pre-rendered and indexable by design. Every Disallow rule should have a specific, intentional reason.
- Do NOT block CSS, JavaScript, or image assets — static sites rely entirely on these for rendering, and Googlebot needs them to evaluate page experience.
- Do NOT add a broad Disallow: / to a production static site — it removes your entire domain from Google's index within days.
Configure robots.txt for your static site
Test your existing file or generate the right configuration for your platform — free, no account required.
Related robots.txt guides
More platform-specific guides in the series.
Frequently Asked Questions
Common questions about robots.txt on static sites hosted on Netlify, Vercel, and similar platforms.
Do static sites need robots.txt?
Yes — all publicly accessible websites benefit from a robots.txt file. For static sites, the file is especially important for two reasons: (1) it tells crawlers where your sitemap is, enabling faster discovery of all your pre-rendered pages; and (2) it blocks any platform-specific utility paths (like Netlify Functions endpoints or CMS admin areas) that shouldn't appear in search results.
Where do I put robots.txt on Netlify?
Place robots.txt in your site's root output directory — the folder Netlify deploys as your site root (typically /public/ for Gatsby, /out/ for Next.js static export, or the root directory for plain HTML sites). Netlify serves it directly at yourdomain.com/robots.txt. You do not need to configure any special Netlify settings for a plain robots.txt file.
Where do I put robots.txt on Vercel?
For Next.js on Vercel, place robots.txt in /public/ (or use app/robots.ts for Next.js 13.3+). For other frameworks (Astro, SvelteKit, Nuxt), place it in the framework's static files directory (e.g. /static/ for SvelteKit, /public/ for Astro). Vercel serves it automatically at yourdomain.com/robots.txt. No special Vercel configuration is required.
How do I generate a sitemap for a static site?
Most static site generators have built-in or plugin-based sitemap generation: Gatsby uses gatsby-plugin-sitemap, Astro includes a built-in @astrojs/sitemap integration, Hugo generates sitemap.xml by default, and Jekyll has jekyll-sitemap. For plain HTML static sites, generate a sitemap.xml file manually or with a build script. Reference the output URL in your robots.txt Sitemap: directive.
Should I block /.netlify/ in robots.txt?
Yes. The /.netlify/ path is used by Netlify for internal functions and API endpoints. These return JSON or function responses, not web pages, so they have no value in search results. Blocking them prevents crawlers from wasting resources on non-indexable content. If you use Netlify Identity or other admin features, those paths should also be blocked.
For more configuration examples, see our robots.txt examples and best practices guide.