Crawler directory
About this crawler
Generic xAI crawler token seen on product fetches.
Usually means: This operator is fetching the page for a product workflow — ads, previews, or internal tools — not a live answer and not a training crawl.
User agent
xAI-Bot
Trfyx matches these as user-agent tokens, not full browser strings. Providers may add surrounding version or product text.
How Trfyx matches
Matched from the user-agent token
Trfyx reads the request on the server, because these crawlers ask for HTML without running your analytics script.
User agent
Request user-agent contains xAI-Bot.
Server fetch
Recorded in middleware, not the browser script.
robots.txt
The published name to allow or disallow.
Sample robots.txt
robots.txt
User-agent: xAI-Bot
Disallow: /See when xAI-Bot
hits your site
Trfyx records xAI-Bot on the server and keeps it out of your human visitor counts.
20-day free trial · No card requiredEU-hosted cloud
EU-hosted cloud