Check What Google and AI Crawlers See in Your Website HTML
Search engines and AI tools do not see your website the way you do. They download the HTML and read what is in it. If a page blocks them, sends them something different, or only shows its words after JavaScript runs, they may get far less than your visitors. Enter any URL below to see exactly what Googlebot, Bingbot and the main AI crawlers from OpenAI, Anthropic and Perplexity receive, side by side with a normal visitor.
I work with Wix businesses across the UK, US, Canada, Australia and worldwide.
This tool sends requests using each crawler's published user agent from our server. It is not the real crawler. Some sites check crawler IP addresses and may treat these requests differently from the real thing, so a block here does not always mean the real crawler is blocked, and the other way round.
The tool reads the HTML as delivered and does not run JavaScript.
What this checker shows you
- The HTTP status each crawler gets, and any redirects
- Whether robots.txt allows or blocks each crawler
- The title, meta description, H1 and canonical each crawler sees
- How many words of real text are in the HTML
- Which structured data (JSON-LD) is present
- Whether any crawler gets different content from a visitor
- Whether your site has an llms.txt file
Why the HTML matters for AI answers
AI assistants such as ChatGPT, Claude and Perplexity can only quote and recommend what their crawlers manage to read. Google can render JavaScript, but it does so in a later step, after crawling. AI crawlers may not run JavaScript at all. Having your key words, headings and structured data in the delivered HTML is the safest way to make sure every crawler can understand the page.
Related lessons and tools
- How Google crawls, indexes and ranks websites
- Monitoring AI crawler traffic on your Wix site
- llms.txt and ai.txt files for AI crawlers
- noindex, canonical and robots directives explained
- Redirects, crawl errors and robots.txt on Wix
- Free robots.txt generator
- Free schema validator
- Free heading checker
- Free AI visibility checker
Found a problem you are not sure how to fix? My Wix SEO Fix Pack gives you the exact fixes for any page, ready to paste into Wix.
Frequently Asked Questions
Is this tool free?
Yes. It is free to use, with no sign-up. There is a limit of 10 checks per hour to keep it fair for everyone.
Is this exactly what the real Googlebot or GPTBot sees?
It is very close, but not identical. The tool uses each crawler's published user agent from our own server. Some websites check the crawler's IP address as well, so they may treat the real crawler differently. Use it as a strong guide, then confirm Google's view with the URL Inspection tool in Google Search Console.
Why does Google see my content but an AI crawler sees almost nothing?
Google renders JavaScript in a later step, but AI crawlers may not run JavaScript at all. If your words only appear after scripts load, an AI crawler may get an almost empty page. Make sure key content is in the HTML.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
OpenAI says GPTBot collects content to improve its AI models, OAI-SearchBot finds websites to show in ChatGPT search, and ChatGPT-User visits pages when a ChatGPT user asks it to. You can allow or block each one separately in robots.txt.
Should I block AI crawlers in robots.txt?
It depends on your goals. Blocking training crawlers such as GPTBot keeps your content out of model training, but blocking search crawlers such as OAI-SearchBot or PerplexityBot can stop your site appearing and being linked in those AI answers. Many businesses block training but allow search.
What is llms.txt, and do I need one?
llms.txt is a proposed plain-text file that gives AI tools a short guide to your site. It is not an official standard, and no major AI company has said it relies on it, so treat it as optional. This tool simply tells you whether you have one.
By Michael Andrews - Wix Marketplace Legend Partner | Get a free audit from Michael