Security & SEO

Robots.txt Generator

Control exactly who accesses your site. Block AI scrapers while keeping search engines happy. Protect your content from unauthorized training data.

Crawler Permissions

Googlebot Allowed

Google's main crawler for search.

GPTBot Blocked

OpenAI's crawler for training ChatGPT.

CCBot Blocked

Common Crawl - used by many AI models.

Bingbot Allowed

Microsoft Bing's search crawler.

Google-Extended Blocked

Google's AI training crawler (Gemini).

Anthropic-AI Blocked

Crawler for Claude AI models.

Applebot Allowed

Apple's crawler for Siri & Spotlight.

FacebookBot Allowed

Meta's crawler for link previews.

Disallowed Paths

/admin//private//wp-admin/

Sitemap Location

robots.txt

# Global Rules
User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /wp-admin/
Allow: /

# Block GPTBot
User-agent: GPTBot
Disallow: /

# Block CCBot
User-agent: CCBot
Disallow: /

# Block Google-Extended
User-agent: Google-Extended
Disallow: /

# Block Anthropic-AI
User-agent: Anthropic-ai
Disallow: /

Warning: Incorrectly configuring robots.txt can de-index your entire site. Always test using Google Search Console's Robots Testing Tool before deploying.