Robots.txt for Laravel (Example, Template & Best Practices)
QUICK SUMMARY
- Place
robots.txtin Laravel's/public/directory — the web root - Always block
/storage/and/vendor/to prevent indexing of application internals - Block
/telescope/and/horizon/— debug and queue dashboard tools - Use
spatie/laravel-sitemapfor automatic sitemap generation - Use
APP_ENVto serve different robots.txt for staging vs production
USEFUL TOOLS
A robots.txt for Laravel must be placed in the /public/ directory — Laravel's web root. Unlike frameworks where the project root is the document root, Laravel deliberately separates application code (in the root) from publicly served files (in /public/). Getting this right is the first step. From there, blocking Laravel-specific paths like /storage/, /vendor/, and debug tools like Telescope and Horizon keeps your crawl budget focused on the pages that matter.
Laravel robots.txt template
This template covers the most important Laravel-specific paths to block. Place it at public/robots.txt and update the Sitemap URL to match your production domain. You can also use our robots.txt generator to customise and download it:
User-agent: *
Disallow: /admin/
Disallow: /storage/
Disallow: /vendor/
Disallow: /telescope/
Disallow: /horizon/
Disallow: /*.json$
Sitemap: https://yourdomain.com/sitemap.xmlWhat each rule does
- /admin/: Your Laravel admin panel (Filament, Nova, or custom routes). Should always be blocked from indexing regardless of authentication.
- /storage/: Contains application logs, compiled Blade views, and the framework cache. Not a web-accessible directory by default, but blocking it in robots.txt is a good defensive measure.
- /vendor/: Composer dependencies — thousands of PHP files that are never meant to be web-accessible. Blocking this path prevents accidental exposure via misconfigured servers.
- /telescope/: Laravel Telescope exposes detailed request logs, database queries, and job data. Never let this appear in search results — block it in robots.txt and protect it with middleware.
- /horizon/: Laravel Horizon's queue monitoring dashboard. Contains job status, failed job logs, and throughput metrics. Should be blocked and authenticated.
- /*.json$: Blocks any
.jsonfiles served from your public directory from being indexed. JSON responses have no value in search results.
Once deployed, test this robots.txt using our robots.txt tester to confirm your public pages are accessible and all sensitive paths are blocked. For a broader reference, see our robots.txt examples library.
Common Laravel robots.txt mistakes
These errors are specific to Laravel's file structure and can quietly expose application internals.
Placing robots.txt in the project root instead of /public/
Laravel's document root is /public/ — files in the project root are never served to web visitors. Always place robots.txt inside /public/robots.txt.
Forgetting to block /telescope/ and /horizon/ on production
These Laravel debug tools expose detailed application internals. Both should be blocked in robots.txt and protected by authentication middleware. Never deploy Telescope to production without proper access controls.
Not blocking /storage/ when using the default filesystem
Laravel logs, compiled views, and framework cache files live in /storage/. Even though they're rarely served via the web, blocking the path in robots.txt prevents accidental exposure.
Missing the Sitemap directive
Laravel doesn't generate a sitemap by default. Use spatie/laravel-sitemap to generate one and always reference it in the Sitemap: line of your robots.txt.
Laravel robots.txt best practices
Follow these guidelines to keep your Laravel application's crawl setup clean and secure.
Always place robots.txt inside /public/
This is Laravel's web root and the only directory accessible to web visitors. The project root where composer.json lives is never served directly.
Block all Laravel debug and admin tools in production
Telescope, Horizon, and any custom /admin/ route should be blocked. These tools expose application internals and have no SEO value.
Use spatie/laravel-sitemap for automatic sitemap generation
This well-maintained package integrates directly with Laravel's routing and Eloquent models to generate comprehensive sitemaps that you can reference in robots.txt.
Use environment-aware robots.txt for staging environments
Return Disallow: / for non-production APP_ENV values. This prevents accidental indexing of staging content without manually managing separate files.
When NOT to use these rules
Applying these directives incorrectly can expose files or break public-facing content. Keep these limits in mind:
- Do NOT place robots.txt in Laravel's project root — only files in /public/ are accessible to the web server.
- Do NOT block /public/storage/ if it contains product images or media that should appear in search results — that's the public filesystem symlink.
- Do NOT rely solely on robots.txt to protect /telescope/ and /horizon/ — always add authentication middleware on the routes themselves.
Configure robots.txt for your Laravel application
Test your existing configuration instantly or build the correct production file from scratch — free, no login required.
Related robots.txt guides
More platform-specific guides in the series.
Frequently Asked Questions
Common questions about robots.txt configuration in Laravel applications.
Where does robots.txt go in a Laravel project?
Place robots.txt in Laravel's /public/ directory. Laravel's web root is /public/ — this is what your Apache or Nginx server points to, so files placed here are served directly at yourdomain.com/filename. A robots.txt in the project root (where composer.json lives) will not be accessible to the web server.
Should I block /storage/ in Laravel?
Yes — with one important caveat. The /storage/ directory contains Laravel's application logs, compiled Blade views, and the framework cache. However, if you use Laravel's /storage/app/public/ filesystem for user-uploaded files (linked via php artisan storage:link), those files are served from /storage/app/public/ via a symlink in /public/storage/. Block /storage/ in robots.txt, but be aware that /public/storage/ (symlinked uploads) may still be accessible and may need separate handling.
How do I generate a sitemap in Laravel?
The most popular approach is the spatie/laravel-sitemap package. Install it via Composer, define which URLs to include, and generate the sitemap as part of your deployment process or as a scheduled artisan command. Reference the output URL in the Sitemap: directive of your robots.txt file.
Should I block /telescope/ and /horizon/ in production?
Yes — always. Laravel Telescope is a debugging and monitoring tool that exposes detailed request data, SQL queries, and application logs. Laravel Horizon is the Redis queue dashboard. Both should be blocked in robots.txt AND properly protected by authentication middleware on your production server. Never leave these tools publicly accessible.
Can I have different robots.txt for staging and production in Laravel?
Yes. Use your .env file and a dedicated controller or middleware to serve different robots.txt content based on APP_ENV. In staging, return Disallow: / to block all crawlers. In production, return your full optimised robots.txt. You can also use Nginx/Apache configuration to swap a static file based on the environment, or set up separate robots.txt files in your deployment pipeline.
For more configuration examples, see our robots.txt examples and best practices guide.