Wix AI bot blocking: controlling AI crawler access via the Robots.txt Editor

Wix gives you native control over which AI crawlers can access your site through the built-in Robots.txt Editor. With the explosion of AI crawlers in 2026, managing this access has become a critical part of your SEO strategy. This lesson walks you through Wix's specific tools for blocking or allowing AI bots, monitoring their activity, and making strategic decisions about AI access to your content.

The Wix Robots.txt Editor for AI Bot Management

Accessing and configuring the editor

• In your Wix dashboard, go to Marketing & SEO then SEO Tools
• Click on Robots.txt Editor under the Tools section
• You will see the default Wix robots.txt with standard directives
• Add new User-agent blocks for each AI crawler you want to manage
• Save your changes and they will be live immediately at yoursite.com/robots.txt
• Verify your changes by visiting yoursite.com/robots.txt in a browser

Which AI Bots to Block and Which to Allow

Not all AI crawlers serve the same purpose. Some crawl exclusively for model training (taking your content to improve their AI), while others crawl to provide real-time answers that cite your website. Blocking training-only crawlers protects your content from being absorbed into AI models. Allowing retrieval crawlers means your business appears when potential customers ask AI tools for recommendations.

• GPTBot: training crawler for OpenAI models. Block if you want to protect content from AI training.
• ChatGPT-User: real-time browsing agent. Allow to appear in ChatGPT answers when users search.
• ClaudeBot: training crawler for Anthropic. Block if protecting proprietary content.
• Google-Extended: Gemini AI training. Block to prevent Google AI training. Does not affect regular Google search.
• CCBot: Common Crawl dataset. Block to prevent open-source AI training.
• PerplexityBot: retrieval crawler. Allow to appear in Perplexity AI search answers.
• Applebot-Extended: Apple Intelligence training. Block to prevent Apple AI model training.
• Bytespider: ByteDance AI training. Block unless you target the Chinese market.

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

# Allow AI retrieval crawlers (so your content appears in AI answers)
User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

Monitoring AI Bot Activity After Changes

After configuring your robots.txt, monitor the Wix Bot Traffic dashboards to verify that blocked crawlers have stopped visiting and allowed crawlers continue. It may take 1-2 weeks for all AI crawlers to respect new robots.txt directives, as they revisit the file on their own schedule. Some crawlers are better at respecting robots.txt than others, and a small number may ignore it entirely.

Selective Blocking: Protecting Specific Content

Instead of blocking AI crawlers site-wide, you can protect only specific directories or page types. For example, if you have a proprietary research section but want your service pages discoverable by AI, you can allow AI crawlers on most of your site while blocking specific paths.

User-agent: GPTBot
Disallow: /blog/research/
Disallow: /premium-content/
Allow: /services/
Allow: /about/

How to Edit Your Wix Robots.txt to Block Training Crawlers While Allowing Retrieval Bots

How to configure the Wix robots.txt editor to allow AI retrieval bots while blocking training-only crawlers from your site

• Log in to your Wix dashboard at manage.wix.com and navigate to Marketing & SEO in the left sidebar.
• Click SEO Tools and then look for the Robots.txt Editor option. Click it to open the editor.
• Review the current robots.txt content. Note the default Wix directives and identify any User-agent blocks that may already be present.
• At the bottom of the existing content, add a new block for GPTBot. Type User-agent: GPTBot on one line, then Disallow: / on the next line. This blocks OpenAI's training crawler from all pages.
• Add a second block for ClaudeBot with Disallow: / to block Anthropic's training crawler.
• Add a third block for Google-Extended with Disallow: / to prevent your content from being used in Gemini AI training without affecting regular Google Search indexing.
• Add a fourth block for CCBot with Disallow: / to block the Common Crawl dataset used by many open-source AI models.
• Now add Allow blocks for retrieval bots. Add User-agent: ChatGPT-User followed by Allow: / to permit ChatGPT's real-time browsing agent to access your content for answering user queries.
• Add User-agent: PerplexityBot followed by Allow: / to allow Perplexity AI's search crawler to access your content for its citation-based answers.
• Save your changes. Verify the file is live by opening yoursite.com/robots.txt in a browser. Confirm the new User-agent blocks appear exactly as typed with no formatting errors.