Can AI Crawlers Read My Website? How to Check in 15 Minutes

The short version

Before an AI assistant can cite your website, its crawler has to be allowed in and has to be able to read your text. Many businesses fail one of those two tests without knowing it. The good news is that you can check both in about 15 minutes.

Know which crawlers matter

AI companies run more than one crawler, and they do different jobs.

• OpenAI separates search from training. According to OpenAI's crawler documentation, OAI-SearchBot is used to surface websites in ChatGPT's search features, while GPTBot relates to training its models. The two settings are independent, so you can allow search while blocking training.
• Anthropic documents its crawler, ClaudeBot, and how site owners can control it, in its support center. Check that page for Anthropic's current list of agents.

If your goal is to be cited, the search and user-request crawlers are the ones you must not block.

Test 1: Check your robots.txt

Open yourdomain.com/robots.txt in a browser. Look for:

1. A blanket Disallow: / under User-agent: *, which blocks everyone.
2. Rules naming OAI-SearchBot, GPTBot, ClaudeBot or other AI agents.
3. Rules a developer or plugin added years ago that nobody remembers.

Make sure each rule says what you actually intend.

Test 2: Check whether your text exists without JavaScript

This is the problem we see most often on modern website builders. Many sites send the browser an almost empty page, then use JavaScript to fill in the words. People see a normal page. A crawler that does not run JavaScript sees almost nothing.

To test it:

1. Open an important page, such as a service page or article.
2. Right-click and choose View Page Source (not Inspect).
3. Search the source for a sentence you can see on the page.

If the sentence is in the source, crawlers can read it. If it is missing, your content depends on JavaScript, and you should look at prerendering or server-side rendering for your key pages.

Test 3: Check your firewall and bot settings

Security tools and CDNs can block AI crawlers before robots.txt is ever read. If you use a service such as Cloudflare, review its bot and AI-crawler settings to make sure they match your robots.txt.

Test 4: Look at your server logs

If you have access to logs, search them for the crawler names above. A site that wants to appear in ChatGPT search should see OAI-SearchBot visits over time. Because user-agent strings can be faked, OpenAI publishes its crawlers' IP ranges so you can verify real visits.

What to do next

Crawl access is the entry ticket, not the whole game. Once crawlers can read you, the content itself has to answer real questions clearly. Read why your business may not be appearing in ChatGPT for the rest of the checklist.

Sources & further reading

• Overview of OpenAI Crawlers, OpenAI
• Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic