Crawlers Trfyx checks
Every AI assistant, search index, and training crawler classified on the server — because those bots request HTML without running your analytics script.
Search indexes
Discovery / 13 crawlers
OpenAI
OAI-SearchBot
Indexes public pages so they can appear in ChatGPT search.
Anthropic
Claude-SearchBot
Discovers and refreshes pages for Claude search.
Perplexity
PerplexityBot
Keeps Perplexity’s answer and search index fresh.
Google
Google-InspectionTool
Fetches a URL when someone inspects it in Google’s webmaster tools.
Google
Googlebot
Indexes the web for Google Search, including AI-related search features.
Mistral
MistralAI-Index
Indexes public pages for Mistral search and answers.
Microsoft
Bingbot
Indexes the web for Bing Search and Copilot retrieval.
Microsoft
msnbot
Legacy Microsoft crawler still seen on some search fetches.
Amazon
Amzn-SearchBot
Makes pages eligible for Amazon search experiences.
Moonshot AI
Kimi-SearchBot
Indexes public pages for Kimi search.
ByteDance
TikTokSpider
Indexes public pages for TikTok search surfaces.
Baidu
Baiduspider
Indexes the web for Baidu Search.
You.com
YouBot
Indexes public pages for You.com search and answers.
See which of these
hit your site
Trfyx splits these crawlers from people, with the pages each one requested. One server-side package.
EU-hosted cloud