search indexes
Googlebot
Operated by Google · Search indexes
Google crawler used for Search indexing and discovery. Cloudflare notes that Googlebot traffic can also support AI-related search features such as AI Overviews.
When Googlebot shows up in your logs
Google is indexing your page for Search, which can also feed AI Overviews.
Discovery crawlers that keep a search or answer index fresh.
Allow or block it in robots.txt
A robots.txt entry is a request. Well-behaved crawlers honor it; it is not an enforcement mechanism.
More from Google
Is Googlebot on your site right now?
Voris AI visibility answers that: verified AI agent traffic, split from your human visitors, on every plan. Cookieless, with a free plan and a 14-day trial that needs no card.