How to manage AI crawlers in your robots.txt

Which AI crawlers exist?

Major AI companies identify their crawlers with specific user-agent strings, grouped here by intent. Because robots.txt rules are per user-agent, you can block training crawlers while keeping search and retrieval crawlers allowed, so your site stays eligible for citation in AI answers. agentmarkup recognizes the following crawlers out of the box:

You can also add custom crawler names for any bot not in the built-in list. Note that some crawlers, such as Bytespider and xAI/Grok agents, comply poorly with robots.txt, so directives are not a guarantee.

Configuration

Set each crawler to 'allow' or 'disallow'. Only configure the crawlers you care about. Missing crawlers are not added to your robots.txt.

1234567891011agentmarkup({
  site: 'https://example.com',
  name: 'My Website',
  aiCrawlers: {
    GPTBot: 'allow',
    ClaudeBot: 'allow',
    PerplexityBot: 'allow',
    'Google-Extended': 'allow',
    CCBot: 'disallow',
  },
})

How it works

agentmarkup uses marker comments to manage its section of your robots.txt. If you already have a robots.txt, the plugin patches it without touching your existing rules. If you do not have one, it creates a new file.

12345678910111213141516171819202122User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

# BEGIN agentmarkup AI crawlers
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Disallow: /

# END agentmarkup AI crawlers

The markers (# BEGIN agentmarkup AI crawlers / # END agentmarkup AI crawlers) allow the plugin to update its rules on every build without duplicating entries or breaking your custom rules.

Conflict detection

If your existing robots.txt has a User-agent: * with Disallow: /, and you configure a crawler to be allowed, agentmarkup warns you about the conflict during build. A broad disallow rule overrides specific allow rules for most crawlers.

This validation catches a common mistake: you intend to allow GPTBot but your existing robots.txt blocks all bots. Without this check, your allow directive would have no effect.

Should you allow or block AI crawlers?

This is a business decision, not a technical one. Consider:

• Allow if you want your content to appear in AI-generated answers, search summaries, and chatbot responses
• Disallow if you do not want your content used for AI model training or AI-powered search results
• Selective access: Allow some crawlers (like PerplexityBot for search) while blocking others (like CCBot for training data)

Combined with llms.txt, JSON-LD structured data, and markdown mirrors, crawler access is one part of a machine-readable website instead of a standalone fix.