Crawler directory
About this crawler
Robots token controlling Gemini training and grounding use.
Usually means: This is a robots.txt token, not a separate HTTP crawler. It tells Google how already-crawled content may be used.
User agent
Google-Extended
Trfyx matches these as user-agent tokens, not full browser strings. Providers may add surrounding version or product text.
How Trfyx matches
Controlled as a robots.txt token
Google-Extended is not its own HTTP crawler. Trfyx files it next to Google’s live bots so you can see the published opt-out.
User agent
Published as the Google-Extended robots token.
Server fetch
Recorded in middleware, not the browser script.
robots.txt
This token is the opt-out.
Sample robots.txt
robots.txt
User-agent: Google-Extended
Disallow: /See when Google-Extended
hits your site
Trfyx records Google-Extended on the server and keeps it out of your human visitor counts.
20-day free trial · No card requiredEU-hosted cloud
EU-hosted cloud