Crawlers Trfyx checks

Fetches a page when a person asks ChatGPT to open it.

Fetches a page when a person asks Claude to open it.

Fetches a page so Perplexity can cite it in a person’s answer.

User-triggered Google fetcher used by AI and product experiences.

Fetches sources a person added to NotebookLM.

Fetches a page for Google read-aloud and assistant playback.

Alternate Google agent token used by AI product fetches.

Fetches a page when a person asks a Mistral product to open it.

Fetches a page when Microsoft Copilot needs it for a person.

Fetches a page when an Amazon assistant needs a live answer.

Fetches sources for DuckDuckGo’s AI-assisted answers.

Fetches pages xAI uses for Grok search-style answers.

Fetches sources during a Grok DeepSearch run.

Fetches a URL a person requested or shared in a Meta product.

Fetches a page when a person asks Kimi to open it.

Fetches a page when a person asks Qwen to open it.

Indexes public pages so they can appear in ChatGPT search.

Discovers and refreshes pages for Claude search.

Keeps Perplexity’s answer and search index fresh.

Fetches a URL when someone inspects it in Google’s webmaster tools.

Indexes the web for Google Search, including AI-related search features.

Indexes public pages for Mistral search and answers.

Indexes the web for Bing Search and Copilot retrieval.

Legacy Microsoft crawler still seen on some search fetches.

Makes pages eligible for Amazon search experiences.

Indexes public pages for Kimi search.

Indexes public pages for TikTok search surfaces.

Indexes the web for Baidu Search.

Indexes public pages for You.com search and answers.

Collects public web text that may be used to train OpenAI models.

Collects public content that may be used to improve Claude.

Generic Google fetcher used by product teams for public pages.

Robots token controlling Gemini training and grounding use.

Crawls sites that owners have connected to Vertex AI agents.

Robots token controlling whether Apple may use Applebot crawls for AI.

Apple’s crawler for Search and, unless opted out, Apple Intelligence.

Crawls public pages to improve Amazon products and services.

Crawls public pages to improve Meta products and AI systems.

Collects public content for Moonshot AI / Kimi systems.

ByteDance’s public-web crawler for product and model data.

Collects public content for Baidu’s ERNIE models.

Collects public content for Alibaba’s Qwen models.

Collects public content for Zhipu AI’s ChatGLM systems.

Collects public content for DeepSeek models.

Cohere crawler for public content used in Cohere systems.

Cohere crawler that collects public pages for training data.

Finds public documents for Allen Institute research systems.

Builds the open Common Crawl datasets that many models train on.

Fetches landing pages for OpenAI advertising surfaces.

xAI crawler associated with Grok product fetches.

Generic xAI crawler token seen on product fetches.

Alternate Grok crawler token from xAI.

xAI’s general web crawler token.

Short Grok token xAI still sends on some requests.

Meta crawler traffic associated with Facebook bot fetches.

ByteDance crawler associated with Doubao product fetches.

Baidu crawler associated with Ernie Bot / Yiyan product fetches.

Alibaba crawler associated with Tongyi product fetches.

Alibaba Cloud crawler seen on product and research fetches.