Should I block GPTBot and ClaudeBot?

Depends on strategy. Blocking stops AI training. Allowing increases citations in ChatGPT, Gemini and Perplexity. Most commercial sites now allow them.

Can robots.txt hide a page from Google?

No. Disallow prevents crawling but not indexing. To prevent indexing, use a noindex meta tag and allow crawling so Google can see the directive.

What should I disallow on a Wix site?

Admin paths, checkout confirmation pages, internal search results and duplicate parameterised URLs. Never disallow real content.

Does robots.txt affect ranking?

Indirectly. Blocking crawlers from real content crashes rankings. Optimising robots.txt to focus crawl budget on high-value pages boosts indexation quality.

Can I have multiple sitemap lines?

Yes. List every sitemap (sitemap.xml, blog-sitemap.xml, image-sitemap.xml) on separate Sitemap: lines.

A non-standard directive some crawlers (Bing, Yandex) respect to throttle crawl rate. Google ignores it. Only use if your server is overloaded.

Why is my Wix robots.txt not updating?

Wix caches robots.txt for 24 hours. If you edit it and do not see changes, clear browser cache or wait a day.

Free Robots.txt Tool

Robots.txt Generator

Create a properly formatted robots.txt file to control how search engines crawl and index your website. Protect sensitive pages, manage bot access, and optimise your crawl budget effectively.

Wix SEO Course

About the Robots.txt Generator

Generate a valid robots.txt file for any Wix site. Control what Google, Bing and AI crawlers (GPTBot, ClaudeBot, PerplexityBot) can and cannot fetch.

Quick summary

Pre-built rules for Googlebot, Bingbot and 10+ AI crawlers
Wix-specific disallow patterns for admin, checkout and editor paths
Includes sitemap declaration line
Validates syntax against Google robots spec
Copy-paste ready for Wix Dashboard > SEO Tools

Use the tool below

Configuration

User Agent

Use * for all search engines or specify: Googlebot, Bingbot, etc.

Allow all pages to be crawled

When checked, search engines can crawl all pages (recommended for most websites)

Disallow Specific Paths (optional)

Enter one path per line (e.g., /admin/, /private/)

Crawl Delay (seconds, optional)

Time to wait between requests (use only if your server needs it)

Sitemap URL (optional)

Help search engines find your sitemap

Common Use Cases

•Block admin panels: /admin/
•Block search results: /search/
•Block API endpoints: /api/
•Block temporary files: /tmp/

Generated robots.txt

Upload to your website root directory

# robots.txt generated by WixSEO
# https://wixseo.co.uk/wix-seo-expert-uk-tools/robots-generator

User-agent: *
Allow: /

How to Use

1.Download or copy the generated robots.txt file
2.Upload it to the root directory of your website
3.Verify it's accessible at: yoursite.com/robots.txt
4.Test it using Google Search Console

Important Notice

The robots.txt file does NOT guarantee that pages won't be indexed. For sensitive content, use password protection or noindex meta tags instead.

How to use this tool

1
Choose presets
Pick from Allow All, Block AI, Wix Defaults, or custom.
2
Add custom disallow rules
Add paths like /admin/ or /checkout/ you want blocked from crawlers.
3
Add sitemap line
Add your sitemap.xml URL. Wix auto-generates this at yoursite.com/sitemap.xml.
4
Deploy in Wix
Open Wix Dashboard > SEO Tools > robots.txt and paste in the generated rules. Publish.

Examples & benchmarks

Wix default

User-agent: * / Allow: /

Safe starting point

Block GPTBot

User-agent: GPTBot / Disallow: /

Blocks ChatGPT training

Frequently asked questions

Yes. Every Wix site has a default robots.txt. You can override it via Wix Dashboard > SEO Tools > robots.txt Editor.

More Free SEO Tools

Backlink Checker

Analyse your backlink profile

Meta Tag Generator

Create optimised meta tags

Sitemap Generator

Generate XML sitemaps

Quick Win Calculator

Find your easiest SEO wins

Accessibility Checker

Check WCAG compliance

Heading Checker

Audit heading structure & hierarchy