search indexes
Googlebot
Operated by Google · Search indexes
Google's main search crawler, indexing the web for Search. Cloudflare notes the same traffic can feed AI Overviews.
When Googlebot shows up in your logs
Google is indexing your page for Search, which can also surface it in AI Overviews.
Discovery crawlers that keep a search or answer index fresh.
Allow or block it in robots.txt
A robots.txt entry is a request. Well-behaved crawlers honor it; it is not an enforcement mechanism.
More from Google
Is Googlebot on your site right now?
Voris AI visibility answers that: verified AI agent traffic, split from your human visitors, on every plan. Cookieless, with a free plan and a 14-day trial that needs no card.