AI & Search Wiki Entry

AI Crawlers

Bots from AI companies that crawl the web to gather training data and index content.

Major AI crawlers include GPTBot (OpenAI), Claude-Web (Anthropic), PerplexityBot, Google-Extended (Gemini), and Bingbot. Site owners can selectively allow or block these crawlers via robots.txt. Blocking AI crawlers prevents inclusion in their training and citation systems but may limit AI search visibility.

Related Entries

robots.txt

A text file at the site root that instructs crawlers which URLs to access.

GPTBot

OpenAI's web crawler that gathers content for ChatGPT training and search.

AI Overview

Google's AI-generated answer summary appearing at the top of search results.

Want to apply this in practice?

The Complete Wix SEO Course covers this topic and 67 others in step-by-step lessons designed for real Wix sites.

Explore the course