Robots.txt: Block Bots Without Breaking Your Site

# Robots.txt: Block Bots Without Breaking Your Site

Your website's robots.txt file is the first thing crawlers look for. Get it wrong and you might accidentally block Google from indexing your best pages — or let AI training bots scrape your content for free.

What Robots.txt Actually Does

Robots.txt tells search engine crawlers and bots which parts of your site they're allowed to visit. It's a voluntary standard — good bots like Googlebot respect it, but malicious scrapers ignore it entirely.

A basic robots.txt looks like this:

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Sitemap: https://example.com/sitemap.xml

Common Mistakes I've Seen

1. Blocking the entire site — Disallow: / blocks everything. I've seen this deployed to production by accident.
2. Forgetting the sitemap line — Crawlers will find your pages eventually, but listing your sitemap speeds up indexing.
3. Blocking CSS and JS files — Google needs to render your pages properly. If you block /assets/ or *.css, Google sees a broken version of your site.

Should You Block AI Training Bots?

This depends on your goals. Blocking GPTBot, CCBot, and Google-Extended prevents your content from being used as training data — but some publishers report losing referral traffic from AI-powered search results.

My take: if you're running a content site that depends on search traffic, blocking AI training bots is reasonable. If you're running a SaaS or e-commerce site, the referral traffic from AI search might be worth more than the training data you're giving up.

How to Generate One

You can write robots.txt by hand, or use our Robots.txt Generator which includes presets for common AI bots and a sitemap reference. It's free and runs in your browser.

## Common robots.txt Patterns

• User-agent: * — Applies to all crawlers
• Disallow: / — Block the entire site
• Disallow: /admin/ — Block a specific directory
• Disallow: /*.pdf$ — Block all PDF files
• Allow: /blog/ — Explicitly allow a directory (useful when blocking a parent)
• Sitemap: https://example.com/sitemap.xml — Point crawlers to your sitemap
• Crawl-delay: 10 — Ask crawlers to wait 10 seconds between requests

robots.txt for Different Search Engines

Major search engines interpret robots.txt slightly differently:

• Google: Fully supports Allow, Disallow, Sitemap, and Crawl-delay (though Google ignores Crawl-delay). Supports pattern matching with * and $.
• Bing: Supports the same directives as Google, and respects Crawl-delay.
• Baidu: Supports basic Allow and Disallow. Pattern matching support is limited.
• Yandex: Has its own extensions, including Clean-param for handling URL parameters.

robots.txt Does Not Guarantee Privacy

A common misconception is that Disallow hides pages from search engines. It does not. It tells crawlers not to visit those pages, but:

1. The URL may still appear in search results if another site links to it
2. Malicious crawlers ignore robots.txt entirely
3. The page is still accessible to anyone who knows the URL

To truly hide a page, use noindex meta tag or HTTP header, or require authentication.