training crawlers
CCBot
Operated by Common Crawl · Training crawlers
Common Crawl crawler that builds open web crawl datasets.
When CCBot shows up in your logs
Common Crawl is archiving your page into its open web dataset, used by many model builders.
Crawlers that collect public content that may be used to train or ground models.
Allow or block it in robots.txt
A robots.txt entry is a request. Well-behaved crawlers honor it; it is not an enforcement mechanism.
Is CCBot on your site right now?
Voris AI visibility answers that: verified AI agent traffic, split from your human visitors, on every plan. Cookieless, with a free plan and a 14-day trial that needs no card.