Pre-Configured Long-Tail Template • 100% Free

Robots.txt Generator: Block GPTBot Training, Allow Perplexity & SearchGPT Citations

Pre-configured policy allowing conversational AI search engines (PerplexityBot, OAI-SearchBot) while blocking foundational scrapers.

Preserve high-intent AI referral traffic and protect intellectual property from uncredited foundation model pre-training.

AI Crawler Directives & Private Routes

Protect training data copyright while preserving visibility and citations in Perplexity and SearchGPT.

Canonical Site Domain

Granular AI Crawler Permissions

OAI-SearchBotSearch Citation

Powers real-time search queries and link citations in ChatGPT with Search.

PerplexityBotSearch Citation

Indexes real-time web pages to generate cited answers and organic search referrals.

ClaudeBotSearch Citation

Crawls pages for Anthropic Claude web search tools and context ingestion.

GPTBotModel Training

Scrapes web content to train future OpenAI foundation models (GPT-5, etc.).

Google-ExtendedModel Training

Allows webmasters to opt out of Google Gemini and Vertex AI training data.

CCBotOpen Scraper

Public web archive crawler utilized by broad open-source LLM datasets.

BytespiderOpen Scraper

Aggressive crawler used for TikTok and Doubao LLM training.

Protected Internal Routes (Disallow List)

/api//admin//workspace//dashboard//internal/

Citation Preservation Strategy

Notice how OAI-SearchBot and PerplexityBot are kept Allow while GPTBot is Disallow. This ensures high-intent organic AI citations without donating free content to train model weights.

Generated robots.txt File

# =============================================================================
# robots.txt generated by CadreGPT Enterprise AI & GEO Generator
# https://cadregpt.com/tools/ai-robots-txt-generator
# =============================================================================

# 1. Classical Search Engines (Google, Bing, Yahoo)
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /workspace/
Disallow: /dashboard/
Disallow: /internal/

# 2. Generative Engine Optimization (GEO) & AI Crawlers

# OAI-SearchBot (OpenAI - Search Citation)
User-agent: OAI-SearchBot
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /workspace/
Disallow: /dashboard/
Disallow: /internal/

# PerplexityBot (Perplexity AI - Search Citation)
User-agent: PerplexityBot
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /workspace/
Disallow: /dashboard/
Disallow: /internal/

# ClaudeBot (Anthropic - Search Citation)
User-agent: ClaudeBot
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /workspace/
Disallow: /dashboard/
Disallow: /internal/

# GPTBot (OpenAI - Model Training)
User-agent: GPTBot
Disallow: /

# Google-Extended (Google - Model Training)
User-agent: Google-Extended
Disallow: /

# CCBot (Common Crawl - Open Scraper)
User-agent: CCBot
Disallow: /

# Bytespider (ByteDance - Open Scraper)
User-agent: Bytespider
Disallow: /

# 3. Discovery Endpoints
Sitemap: https://cadregpt.com/sitemap.xml
# LLM Knowledge Feeds for Frontier Agents
Allow: /llms.txt
Allow: /llms-full.txt

Enterprise Specification Matrix

Manual Approach vs. Governed CadreGPT Enterprise

Why organizations transition from unmanaged scripts to governed autonomous agent workspaces.

Side-by-side technical comparison for Robots.txt Generator: Block GPTBot Training, Allow Perplexity & SearchGPT Citations
Feature Dimension	Manual / Unmanaged Way	Automated with CadreGPT Enterprise
AI Search vs. Training Separation	A blanket Disallow: / blocks Perplexity and SearchGPT citations, eliminating valuable organic AI referral traffic.	Granular rules permitting search citation bots (OAI-SearchBot, PerplexityBot) while blocking scraping bots (CCBot, Bytespider).
Private Route Shielding	Accidentally leaving /admin, /api, or customer workspace URLs indexable in public crawler manifests.	Standardized enterprise disallow lists safeguarding internal routes, authenticated dashboards, and API webhooks.
LLM Document Discovery	AI crawlers waste token crawl budgets on irrelevant asset bundles and pagination query strings.	Explicit Sitemap and llms.txt declarations pointing AI bots directly to high-signal canonical knowledge trees.
Dynamic Policy Updates	Manually editing static robots.txt files whenever new AI frontier labs release unannounced crawler user-agents.	Continuously updated crawler registry tracking user agents from OpenAI, Anthropic, Google, Perplexity, and Apple.

Direct Answers for Humans & Search Engines

Frequently Asked Questions

Technical answers and operational best practices for Block Training, Allow Citations.

01.What is the difference between GPTBot and OAI-SearchBot?

GPTBot scrapes website content to pre-train OpenAI foundation models (like GPT-4 and future models). OAI-SearchBot specifically gathers real-time search context to cite and display links in ChatGPT with Search. Blocking GPTBot does not remove your site from search citations if OAI-SearchBot is allowed.

02.Should I allow or block PerplexityBot?

If you want your website cited as an authoritative source and receive referral traffic from Perplexity AI search queries, you should allow PerplexityBot. Blocking it prevents your brand from surfacing in AI search answers.

03.What is Google-Extended?

Google-Extended allows web publishers to opt out of having their content used to train Google generative AI models (like Gemini) without impacting their standard Google Search indexing and rankings.

04.Where should the robots.txt file be uploaded?

The robots.txt file must reside in the root directory of your domain (e.g., https://yourdomain.com/robots.txt). Search engine and AI bots look exclusively at this root location.