AI crawlers (common tokens)
Publishers often list dedicated User-agent tokens for AI-related fetchers. Tokens and behavior change — treat vendor docs as the source of truth.
- GPTBot — OpenAI crawler token commonly discussed for training / product signals.
- ClaudeBot / anthropic-ai — Anthropic-related tokens.
- PerplexityBot / Perplexity — Perplexity answer-engine crawlers.
- Google-Extended — Gemini / Vertex AI–related preference; not Googlebot search.
- Bytespider — ByteDance crawler often mentioned in AI harvesting discussions.
- CCBot — Common Crawl; datasets may be used downstream for model training.
- Amazonbot, Applebot-Extended — extras publishers frequently review.
Listing a Disallow preference in robots.txt is a signal to cooperative crawlers. It does not ensure any vendor will honor it or refrain from training.