What is robots.txt?

A robots.txt file is a plain text file that tells web robots — most commonly search engines — which pages on your site to crawl and which to ignore. It is the very first thing Googlebot checks when arriving at your domain.

How does robots.txt work?

Before a search engine crawler fetches any page on your site, it looks for a file specifically named robots.txt located at the root of your domain (e.g., https://example.com/robots.txt).

If the file exists, the crawler reads the instructions inside to understand whether it is permitted to access the specific URL it intends to visit. This process is part of the Robots Exclusion Protocol (REP), an industry standard respected by Google, Bing, and most other major crawlers.

Basic Syntax and Directives

A robots.txt file is built from a small set of directives. Each directive is placed on its own line. Here are the four you'll use most often:

1. Allowing all bots full access

User-agent: *
Disallow:

The asterisk * is a wildcard meaning "all crawlers". A blank Disallow line means "disallow nothing" — i.e., allow everything. This is the simplest valid robots.txt example for a fully public site.

2. Blocking the entire site

User-agent: *
Disallow: /

The forward slash / represents the root of the site. This blocks all bots from crawling anything — use it only on staging or development environments, never on a live site.

3. Blocking specific directories

User-agent: Googlebot
Disallow: /admin/
Disallow: /private-data/

This tells specifically Googlebot to stay out of the admin and private-data folders. Browse our robots.txt examples library for more ready-to-use configurations.

Why do you need a robots.txt file?

• Optimising crawl budget: Search engines allocate limited time and resources to crawl your site. Blocking low-value pages — like internal search results, admin panels, or paginated archives — helps Google focus on your important content.
• Keeping private paths out of indexes: While not a security measure, it prevents search engines from indexing sensitive directories that should never appear in public results.
• Preventing server overload: A bot crawling thousands of dynamic parameterised URLs simultaneously can put excessive load on your server. Robots.txt lets you restrict that behaviour.
• Pointing to your sitemap: Including a Sitemap: directive helps every major crawler discover and prioritise your pages more efficiently.

Common SEO misconceptions about robots.txt

1. "I can hide my secret admin page with robots.txt." — No. Bad actors ignore robots.txt entirely. Use password protection or authentication for security. In fact, listing a path in robots.txt can make it more visible to attackers.
2. "Robots.txt removes pages from Google." — No. It stops crawling. If a page is already indexed, you need to allow crawling and add a noindex tag to have it removed from results.
3. "I don't need robots.txt if I have a sitemap." — They serve different purposes. A sitemap tells crawlers which pages to visit; robots.txt tells them which ones to avoid. Both should be used together.

For a full list of what to do and what to avoid, read the robots.txt best practices guide.

Summary

A well-configured robots.txt file is one of the simplest technical SEO wins available to any website owner. It costs nothing, takes minutes to set up, and directly influences how search engines allocate their crawl attention across your site. Whether you're running a simple blog, a large e-commerce store, or a news publication, having a correctly formatted robots.txt file ensures crawlers spend their time on the pages that matter most.

The easiest way to create a robots.txt file without making syntax errors is to use a dedicated tool. Our robots.txt generator gives you a visual builder, live preview, and syntax validator — all in one place, with no login required. Once your file is ready, check your robots.txt file with our free tester to verify every rule before publishing.

Frequently Asked Questions

Answers to the most common questions about the robots.txt file and how it affects your website.