other ai bots
xAI-Web-Crawler
Operated by xAI · Other AI bots
xAI's generically named web crawler. xAI hasn't said what it collects, so consider it undocumented traffic.
When xAI-Web-Crawler shows up in your logs
xAI is crawling your page for something it hasn't described publicly.
AI-related bot traffic that does not fit the categories above.
Allow or block it in robots.txt
A robots.txt entry is a request. Well-behaved crawlers honor it; it is not an enforcement mechanism.
More from xAI
Is xAI-Web-Crawler on your site right now?
Voris AI visibility answers that: verified AI agent traffic, split from your human visitors, on every plan. Cookieless, with a free plan and a 14-day trial that needs no card.