training crawlers
Google-Extended
Operated by Google · Training crawlers
Not a crawler at all, but a robots.txt token that decides whether Google may use your content to train and ground Gemini.
When Google-Extended shows up in your logs
You won't see this in traffic. Set it in robots.txt to allow or refuse Gemini training and grounding.
Crawlers that collect public content that may be used to train or ground models.
Allow or block it in robots.txt
Google-Extended is a policy token: robots.txt is exactly where it lives, and these lines are the whole mechanism.
More from Google
Is Google-Extended on your site right now?
Voris AI visibility answers that: verified AI agent traffic, split from your human visitors, on every plan. Cookieless, with a free plan and a 14-day trial that needs no card.