Start free trial
PricingAI visibilityAI referralsHumans-only analyticsIntegrationsDocsBlogContactAI citation checkerAgent Readiness Scorellms.txt generatorSign in

free reference

The AI crawler directory

Every AI assistant, search index, and training crawler that might visit a website: what it does, whether its traffic can be verified, and how to allow or block it. Built from the providers’ own published sources.

Browse the directoryCheck your site’s readiness
every crawler, one directoryai answers16 crawlerssearch indexes14 crawlerstraining crawlers19 crawlersother ai bots13 crawlersyour site62 crawlers, 21 providers, four intents. reviewed july 2026.
62crawlers documented
21providers, majors to newcomers
4intents: answers, indexing, training, other
July 2026last reviewed against provider sources

Look up any crawler

Search by crawler or provider, or filter by what the bot is for. Every entry links to its own page with verification guidance, sample IP ranges where the provider publishes them, and ready-made robots.txt lines.

All 62 crawlers

ChatGPT-UserAI answers
OpenAI

User-triggered fetches when ChatGPT opens a page to answer a person.

IP-verifiable
Claude-UserAI answers
Anthropic

User-triggered fetches when Claude opens a page to answer a person.

IP-verifiable
Perplexity-UserAI answers
Perplexity

User-triggered fetches for Perplexity answers and source citations.

IP-verifiable
Google-AgentAI answers
Google

Google user-triggered fetcher used by AI or product experiences.

IP-verifiable
Google-NotebookLMAI answers
Google

Google fetcher associated with NotebookLM-style answer workflows.

IP-verifiable
Google-Read-AloudAI answers
Google

Google fetcher associated with read-aloud and assistant experiences.

IP-verifiable
GoogleAgentAI answers
Google

Google user-triggered fetcher used by AI or product experiences.

IP-verifiable
MistralAI-UserAI answers
Mistral

MistralAI-User is tracked as a user-triggered fetcher from Mistral.

IP-verifiable
CopilotAI answers
Microsoft

User-triggered Microsoft/Copilot fetches for AI answers.

IP-verifiable
Amzn-UserAI answers
Amazon

User-triggered Amazon fetches for fresh answers in products such as Alexa.

IP-verifiable
DuckAssistBotAI answers
DuckDuckGo

DuckDuckGo real-time crawler for AI-assisted answers with citations.

IP-verifiable
xAI-SearchBotAI answers
xAI

xAI-SearchBot is tracked as a user-triggered fetcher from xAI.

UA only
Grok-DeepSearchAI answers
xAI

Grok-DeepSearch is tracked as a user-triggered fetcher from xAI.

UA only
meta-externalfetcherAI answers
Meta

Meta fetcher used when a person requests or shares a specific URL.

IP-verifiable
Kimi-UserAI answers
Moonshot AI

Kimi-User is tracked as a user-triggered fetcher from Moonshot AI.

IP-verifiable
Qwen-UserAI answers
Alibaba

Qwen-User is tracked as a user-triggered fetcher from Alibaba.

UA only
OAI-SearchBotSearch indexes
OpenAI

OpenAI crawler used to surface sites in ChatGPT search results.

IP-verifiable
Claude-SearchBotSearch indexes
Anthropic

Anthropic crawler used to discover and refresh content for Claude search.

IP-verifiable
PerplexityBotSearch indexes
Perplexity

Perplexity crawler used to keep its answer and search index fresh.

IP-verifiable
Google-InspectionToolSearch indexes
Google

Google-InspectionTool is tracked as a search or answer-index crawler from Google.

IP-verifiable
GooglebotSearch indexes
Google

Google crawler used for Search indexing and discovery. Cloudflare notes that Googlebot traffic can also support AI-related search features such as AI Overviews.

IP-verifiable
MistralAI-IndexSearch indexes
Mistral

MistralAI-Index is tracked as a search or answer-index crawler from Mistral.

IP-verifiable
BingbotSearch indexes
Microsoft

Microsoft crawler used for Bing Search indexing.

IP-verifiable
msnbotSearch indexes
Microsoft

Legacy Microsoft crawler used for search indexing.

IP-verifiable
Amzn-SearchBotSearch indexes
Amazon

Amazon crawler used to make content eligible for Amazon search experiences.

IP-verifiable
meta-webindexerSearch indexes
Meta

Meta crawler used to improve Meta AI search result quality.

IP-verifiable
Kimi-SearchBotSearch indexes
Moonshot AI

Kimi-SearchBot is tracked as a search or answer-index crawler from Moonshot AI.

IP-verifiable
TikTokSpiderSearch indexes
ByteDance

TikTokSpider is tracked as a search or answer-index crawler from ByteDance.

UA only
BaiduspiderSearch indexes
Baidu

Baiduspider is tracked as a search or answer-index crawler from Baidu.

UA only
YouBotSearch indexes
You.com

YouBot is tracked as a search or answer-index crawler from You.com.

UA only
GPTBotTraining
OpenAI

OpenAI crawler for collecting public content that may improve future models.

IP-verifiable
ClaudeBotTraining
Anthropic

Anthropic crawler for public content that may be used to improve Claude.

IP-verifiable
GoogleOtherTraining
Google

Generic Google crawler used by product teams for public content fetches, including internal research and development.

IP-verifiable
Google-ExtendedTraining
Google

Robots token controlling whether Google may use crawled content for Gemini model training and grounding; it is not a separate HTTP crawler user agent.

robots.txt token
Google-CloudVertexBotTraining
Google

Google crawler used for targeted crawls requested by site owners building Vertex AI agents.

IP-verifiable
Applebot-ExtendedTraining
Apple

Robots token controlling whether Apple may use crawled content for AI training.

robots.txt token
ApplebotTraining
Apple

Apple crawler Cloudflare classifies across search and training use cases.

IP-verifiable
AmazonbotTraining
Amazon

Amazon crawler used to improve Amazon products and services.

IP-verifiable
meta-externalagentTraining
Meta

Meta crawler for indexing or improving Meta products and AI systems.

IP-verifiable
KimiBotTraining
Moonshot AI

KimiBot is tracked as a public-content crawler from Moonshot AI.

IP-verifiable
BytespiderTraining
ByteDance

Bytespider is tracked as a public-content crawler from ByteDance.

UA only
ERNIEBotTraining
Baidu

ERNIEBot is tracked as a public-content crawler from Baidu.

UA only
QwenBotTraining
Alibaba

QwenBot is tracked as a public-content crawler from Alibaba.

UA only
ChatGLM-SpiderTraining
Zhipu AI

ChatGLM-Spider is tracked as a public-content crawler from Zhipu AI.

UA only
DeepSeekBotTraining
DeepSeek

DeepSeekBot is tracked as a public-content crawler from DeepSeek.

UA only
cohere-aiTraining
Cohere

cohere-ai is tracked as a public-content crawler from Cohere.

UA only
cohere-training-data-crawlerTraining
Cohere

cohere-training-data-crawler is tracked as a public-content crawler from Cohere.

UA only
AI2BotTraining
Allen AI

Allen Institute crawler used to find documents for AI research systems.

UA only
CCBotTraining
Common Crawl

Common Crawl crawler that builds open web crawl datasets.

IP-verifiable
OAI-AdsBotOther
OpenAI

OpenAI crawler associated with ad or landing-page fetch workflows.

UA only
GrokBotOther
xAI

GrokBot is tracked as other ai bots from xAI.

UA only
xAI-BotOther
xAI

xAI-Bot is tracked as other ai bots from xAI.

UA only
xAI-GrokOther
xAI

xAI-Grok is tracked as other ai bots from xAI.

UA only
xAI-Web-CrawlerOther
xAI

xAI-Web-Crawler is tracked as other ai bots from xAI.

UA only
GrokOther
xAI

Grok is tracked as other ai bots from xAI.

UA only
meta-externaladsOther
Meta

Meta crawler used for advertising and business product use cases.

IP-verifiable
facebookexternalhitOther
Meta

Meta crawler used for shared link previews on Facebook, Instagram, and Messenger.

IP-verifiable
FacebookBotOther
Meta

Meta crawler traffic associated with Facebook bot fetches.

IP-verifiable
DoubaobotOther
ByteDance

Doubaobot is tracked as other ai bots from ByteDance.

UA only
YiyanBotOther
Baidu

YiyanBot is tracked as other ai bots from Baidu.

UA only
TongyiBotOther
Alibaba

TongyiBot is tracked as other ai bots from Alibaba.

UA only
AliyunBotOther
Alibaba

AliyunBot is tracked as other ai bots from Alibaba.

UA only

A note on freshness: IP ranges shown here are samples from the providers’ published lists as of July 2026. The linked official endpoints are the source of truth; ranges change without notice, and we would rather point you at the live list than pretend a snapshot is one.

Knowing them is half the job

The other half is seeing them on your own site. Voris AI visibility tells humans from AI agents in your traffic, verified, on every plan. Cookieless, with a free plan and a 14-day trial that needs no card.

Start free trialAbout AI visibility