search indexes
Claude-SearchBot
Operated by Anthropic · Search indexes
Anthropic's crawler that indexes pages so Claude can find and cite them when it searches the web.
When Claude-SearchBot shows up in your logs
Anthropic is adding your page to the index Claude reaches for when it searches.
Discovery crawlers that keep a search or answer index fresh.
Allow or block it in robots.txt
A robots.txt entry is a request. Well-behaved crawlers honor it; it is not an enforcement mechanism.
More from Anthropic
Is Claude-SearchBot on your site right now?
Voris AI visibility answers that: verified AI agent traffic, split from your human visitors, on every plan. Cookieless, with a free plan and a 14-day trial that needs no card.