The AI crawler directory

OpenAI's live fetcher. When someone asks ChatGPT a question your page can answer, this is what opens it in the moment.

Anthropic's live fetcher for Claude. It reads your page the instant a person's question calls for it.

The fetch behind a Perplexity answer. It reads your page so Perplexity can quote it and list it as a source.

One of Google's user-triggered agents. It opens a page in the moment, acting for a person rather than fetching an answer.

The fetcher behind Google's NotebookLM. It opens a page when someone adds it as a source to research with.

A Google fetcher for read-aloud and assistant features. It opens your page so the text can be spoken back.

Another label in Google's user-triggered agent family. Like Google-Agent, it fetches pages while carrying out a person's task.

Mistral's live fetcher for Le Chat. It opens your page when a person's question needs it.

Microsoft Copilot fetching live. It reads your page in the moment to answer whatever someone just asked.

Amazon's live fetcher behind products like Alexa, opening your page to pull a fresh answer for someone.

DuckDuckGo's real-time fetcher for DuckAssist. It reads your page so the AI answer can quote and cite it.

xAI's live fetcher for Grok. It opens your page as Grok answers, though xAI publishes no IP ranges to confirm it.

xAI's fetcher for Grok's deep search mode. It reads your page live while building a longer, researched answer.

A Meta fetcher that opens a specific page when a person requests or shares its link inside a Meta product.

Moonshot's live fetcher for Kimi. It opens your page when a question there needs an answer.

Alibaba's live fetcher for Qwen, opening your page mid-answer. Alibaba publishes no IP ranges, so it's user-agent only.

OpenAI's crawler that discovers and refreshes pages so they can turn up in ChatGPT search.

Anthropic's crawler that indexes pages so Claude can find and cite them when it searches the web.

Perplexity's indexing crawler, quietly keeping its answer engine's map of the web current.

Google's inspection fetcher, the one behind Search Console's URL inspection and the Rich Results test.

Google's main search crawler, indexing the web for Search. Cloudflare notes the same traffic can feed AI Overviews.

Mistral's indexing crawler, building the map Le Chat draws on when it answers.

Microsoft's long-running search crawler, indexing pages for Bing and, by extension, Copilot.

A legacy Microsoft search crawler, largely superseded by Bingbot but still turning up in logs.

Amazon's crawler that indexes pages to make them eligible for Amazon's search experiences.

Meta's indexing crawler, working to sharpen the results Meta AI returns in search.

Moonshot's search crawler for Kimi, indexing pages so the assistant can find them when answering.

ByteDance's crawler that indexes pages for TikTok's search and discovery surfaces.

Baidu's main search crawler, indexing pages for China's largest search engine.

You.com's crawler, indexing pages for its search and answer engine.

OpenAI's training crawler. It gathers public pages that may feed a future GPT model.

Anthropic's crawler for gathering public web text that may go toward improving Claude.

A catch-all Google crawler that product teams use to pull public pages for research and development.

Not a crawler at all, but a robots.txt token that decides whether Google may use your content to train and ground Gemini.

Google's crawler for on-demand fetches a site owner requests while building a Vertex AI agent.

A robots.txt token rather than a bot. It tells Apple whether it may use already-crawled content for AI training.

Apple's crawler, which Cloudflare places across both search and training uses, feeding Siri, Spotlight, and Safari.

Amazon's general crawler, gathering public content to improve its products and services.

Meta's crawler that collects public pages to index and improve its AI systems.

Moonshot AI's bot for collecting public content that may train a later Kimi model.

ByteDance's crawler, widely reported to collect public content for model training.

Baidu runs this to collect public pages that may help train its ERNIE models.

Alibaba's crawler for Qwen. It gathers public content that may become training data.

Zhipu AI sends this to gather public content that may go into training ChatGLM.

DeepSeek's crawler for pulling public web content that may feed a future model.

One of Cohere's crawlers, gathering public content for its AI systems.

Cohere's crawler named for its purpose: gathering public content that may become model training data.

The Allen Institute's crawler (Ai2), gathering public documents to feed its AI research systems.

Common Crawl's bot. It archives public pages into a free, open dataset that many model builders later train on.

OpenAI's crawler for ad and landing-page fetches, kept separate from its search and answer bots.

One of xAI's crawlers. xAI has not published what GrokBot fetches, so treat it as unverified xAI traffic for now.

An xAI crawler with no public documentation on what it does. Its role stays unexplained until xAI says more.

A Grok-branded fetcher from xAI, with no published spec. Its exact purpose is unconfirmed.

xAI's generically named web crawler. xAI hasn't said what it collects, so consider it undocumented traffic.

A plain Grok user agent from xAI, undocumented and easy to spoof, so treat it as unverified until it proves itself.

Meta's crawler for advertising and business products, not for answers or search.

The Meta fetcher that builds link previews when someone shares your URL on Facebook, Instagram, or Messenger.

A Meta crawler that turns up outside the usual link-preview flow. Meta files it under its general Facebook fetches.

ByteDance's crawler tied to Doubao, its consumer AI assistant. ByteDance has published little about what it fetches.

Baidu's crawler linked to Yiyan, its AI assistant. Baidu documents it thinly, so specifics are limited.

Alibaba's crawler for Tongyi, its AI assistant family. Alibaba publishes little detail about how it behaves.

A crawler from Alibaba Cloud (Aliyun), with little public documentation about its exact purpose.