Perplexity and Claude bot tracking: monitoring AI crawlers on your Wix site

AI search engines like Perplexity, ChatGPT and Claude send their own crawlers to index web content. Understanding which AI bots visit your Wix site, what they index and how to control their access is the next frontier of search visibility.

The AI Crawler Landscape

Beyond Googlebot, your Wix site is now visited by multiple AI crawlers: GPTBot (OpenAI/ChatGPT), ClaudeBot (Anthropic/Claude), PerplexityBot (Perplexity AI), Bytespider (ByteDance/TikTok), and others. Each crawler indexes your content to power their AI models and search features. Understanding their behaviour helps you optimise for AI search visibility.

Identifying AI Crawlers in Your Logs

• GPTBot: user agent "GPTBot/1.0" from OpenAI, crawls for ChatGPT training and search
• ClaudeBot: user agent "ClaudeBot" from Anthropic, crawls for Claude AI model training
• PerplexityBot: user agent "PerplexityBot" from Perplexity AI, crawls for real-time search answers
• CCBot: user agent "CCBot" from Common Crawl, used by many AI companies
• Bytespider: user agent "Bytespider" from ByteDance, crawls for TikTok and other services

Controlling AI Crawler Access on Wix

You can control which AI crawlers access your Wix site through your robots.txt file. However, the strategic question is whether you should block them. Blocking AI crawlers means your content will not appear in AI-powered search results, potentially losing a growing traffic source. Allowing them means your content may be used to train AI models.

Optimising for AI Crawler Indexing

Making your Wix content AI-crawler friendly

• Ensure your robots.txt allows the AI crawlers you want indexing your content
• Implement LLMs.txt on your Wix site (Module 14) for structured AI communication
• Write clear, factual content that AI systems can accurately extract and cite
• Include source attribution and expertise signals that AI crawlers evaluate for trustworthiness
• Use structured data to give AI crawlers machine-readable context about your content
• Monitor which AI platforms cite your content using brand mention tracking tools
• Track referral traffic from AI search platforms in Google Analytics

How to manage AI crawler access via the Wix robots.txt editor

• Navigate to Dashboard > Marketing & SEO > SEO Tools in your Wix account.
• Click Robots.txt to open the editable robots.txt file for your site.
• Review the existing rules to identify whether any AI crawlers are currently allowed or blocked.
• To block a training-only crawler, add a new User-agent block with "Disallow: /" for the specific bot name (e.g., GPTBot or ClaudeBot).
• To allow a retrieval crawler that powers AI answer results, ensure its user-agent has no Disallow rule or has "Allow: /".
• Add a comment line above each rule using the # character to document why that crawler is allowed or blocked, for future reference.
• Click the Reset to default button if you need to undo all changes and return to the Wix-recommended configuration.
• Save your changes and verify by entering your domain followed by /robots.txt in a browser to confirm the rules are live.
• Set a quarterly reminder to review the robots.txt as new AI crawlers are released regularly.

AI Crawler Tools

• OpenAI GPTBot Documentation: Official GPTBot documentation and robots.txt instructions
• Dark Visitors: Database of AI crawlers and bots with robots.txt rules
• SEO Fix in a Box: My Wix app. Free to install, it scans your site and shows what needs fixing, and paid plans write the fixes from your own Google searches