Crawlers Trfyx checks
Every AI assistant, search index, and training crawler classified on the server — because those bots request HTML without running your analytics script.
Training crawlers
Model data / 19 crawlers
OpenAI
GPTBot
Collects public web text that may be used to train OpenAI models.
Anthropic
ClaudeBot
Collects public content that may be used to improve Claude.
Google
GoogleOther
Generic Google fetcher used by product teams for public pages.
Google
Google-Extended
Robots token controlling Gemini training and grounding use.
Google
Google-CloudVertexBot
Crawls sites that owners have connected to Vertex AI agents.
Apple
Applebot-Extended
Robots token controlling whether Apple may use Applebot crawls for AI.
Apple
Applebot
Apple’s crawler for Search and, unless opted out, Apple Intelligence.
Amazon
Amazonbot
Crawls public pages to improve Amazon products and services.
Meta
meta-externalagent
Crawls public pages to improve Meta products and AI systems.
Moonshot AI
KimiBot
Collects public content for Moonshot AI / Kimi systems.
ByteDance
Bytespider
ByteDance’s public-web crawler for product and model data.
Baidu
ERNIEBot
Collects public content for Baidu’s ERNIE models.
Alibaba
QwenBot
Collects public content for Alibaba’s Qwen models.
Zhipu AI
ChatGLM-Spider
Collects public content for Zhipu AI’s ChatGLM systems.
DeepSeek
DeepSeekBot
Collects public content for DeepSeek models.
Cohere
cohere-ai
Cohere crawler for public content used in Cohere systems.
Cohere
cohere-training-data-crawler
Cohere crawler that collects public pages for training data.
Allen AI
AI2Bot
Finds public documents for Allen Institute research systems.
Common Crawl
CCBot
Builds the open Common Crawl datasets that many models train on.
See which of these
hit your site
Trfyx splits these crawlers from people, with the pages each one requested. One server-side package.
EU-hosted cloud