Is Your Website Ready for AI Agents? I Checked 300 Dutch Websites

For years, your website had one kind of visitor: a person (if we leave search engine crawlers aside for a moment). AI has changed that. AI agents now visit your website looking for information: when ChatGPT, Claude or Perplexity look something up for an answer, they send a bot to fetch your page and read it. With a bit of luck (and good judgement), you are then mentioned or cited in the answers those AI models give.

But that only works if AI agents can access your website. And that is less obvious than it sounds.

AI agents can be prevented from reaching and reading your website in several ways. On top of that, there are a number of methods that are supposed to make your website more attractive to AI bots, such as llms.txt and Markdown versions of your pages. But which of them actually matter?

So I built a free AI Agent Readiness Checker and used it on 300 Dutch websites: 150 that AI platforms cite when people ask questions in Dutch, and 150 picked at random. Below are the results, what they mean for your website, and what I changed on my own.

How do you make your website ready for AI agents?

To be ready for AI agents, a company website or content site has to meet three important conditions:

1. The bots are let in. Neither robots.txt nor the server turns away the bots that fetch pages for AI answers, such as OAI-SearchBot and ChatGPT-User from OpenAI, Claude-SearchBot and Claude-User from Anthropic, and PerplexityBot and Perplexity-User.
2. The content is in the HTML. Research by Vercel and MERJ found that the major AI crawlers don't run JavaScript. Text that only appears after scripts run is invisible to them.
3. Nothing tells them to ignore the page. A noindex tag, a canonical tag pointing elsewhere or a redirect chain can keep a page out of search indexes, and with that out of many AI answers.

Webshops and services that agents should log into or pay through have further work to do, such as payment protocols or an MCP server. For most websites, those don't apply yet.

Bots for AI answers are separate from bots for model training. GPTBot and ClaudeBot collect content to train models. Blocking them doesn't keep you out of ChatGPT or Claude answers; blocking OAI-SearchBot or Claude-SearchBot does. OpenAI, Anthropic and Perplexity each document their bots: OpenAI, Anthropic, Perplexity.

How did I test 300 Dutch websites?

I compared two groups:

• Websites AI cites (150). The domains cited most often in Dutch-language answers in my study of the sources AI uses in the Netherlands, with answers from ChatGPT, Claude, Gemini and Google AI Mode. Mostly companies and brands, comparison sites, webshops and media.
• Random Dutch websites (150). A random sample of .nl domains from the top 300,000 of the Tranco list, a research ranking of popular websites.

In November 2026, the checker fetched each homepage as a normal browser and as eight AI bots, and read robots.txt for those bots plus Googlebot, Bingbot and Google-Extended. I tested 147 of the cited websites and 125 of the random ones; the rest didn't respond or no longer existed. Every server block was checked a second time, and only blocks that happened again were counted.

What did the test show?

*One cited website appeared to block every bot in robots.txt, including Googlebot. When I fetched the file again with a normal browser, it allowed them: the firewall had answered the robots.txt request. I count it as a firewall block, not a robots.txt block.

Hardly any website blocks AI bots in robots.txt

Much of the advice about AI visibility starts with robots.txt. Makes sense: that is where you tell bots what they may do. But is that where things go wrong? Hardly. None of the 147 websites AI cites asked AI answer bots to stay away in their robots.txt. Among the random websites, 6 of 125 (4.8%) did, including two news media, which often block AI bots on purpose.

Training bots were blocked more often: 12% of the random websites blocked GPTBot or ClaudeBot in robots.txt. That is a choice about model training, not about AI search. A few sites blocked training bots and left the answer bots open, which is a sensible way to separate the two.

Google-Extended, the token that controls whether Google may use content for Gemini, was blocked by 9.6% of the random websites and 1.4% of the cited ones. Googlebot itself was almost never blocked.

My advice: Check that your robots.txt doesn't block OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot or Perplexity-User, unless you want to stay out of their answers. If you want to block model training, name GPTBot and ClaudeBot specifically.

Robots.txt says yes, the server says no

The bigger difference was on the server. Robots.txt is a request; the firewall decides what actually happens. I saw two kinds of blocking.

The browser gets in, the AI bot doesn't

On some websites my browser got the page, but one or more AI bots got an error or a security check instead. That points to a rule aimed at AI bots. Among the random websites that let my browser in, this happened on 15 of 89 (17%). On 12 of those, robots.txt allowed the bot that was turned away. The website asked the bots to come in and the server sent them away.

Among the cited websites, it happened on 4 of 117 (3.4%). Which makes sense: if you turn AI bots away, you are less likely to be cited.

These blocks often come from firewalls and bot protection services rather than from a deliberate decision. Cloudflare is the best-known example: since 1 July 2025 it has blocked AI crawlers by default on new domains. About half of the random websites that blocked AI bots on the server ran on Cloudflare, so Cloudflare isn't the whole story. Other hosting firewalls and security plugins do the same.

One in five websites refuses every bot, and AI still cites them

There is also a blunter form of blocking. On 20% of the cited websites and 29% of the random ones, the firewall refused even my normal browser request, because it came from a server rather than a real person. This is general bot protection, common at banks, large webshops and airlines.

The AI bots were usually refused too. But that evidence is weaker: some bot protection lets in the real AI bots, by checking that a request comes from OpenAI's or Anthropic's published addresses. And on 7 cited and 3 random websites, the AI bots got through where my browser didn't.

One in five of the websites AI cites most for Dutch questions turned away every automated request I sent. Yet AI cites them.

How can ABN AMRO be cited if it turns AI bots away?

The clearest example was ABN AMRO. My browser got the page, but all six AI answer bots were refused, on both abnamro.nl and abnamro.com. Robots.txt allowed them. Even so, ChatGPT, Claude and Gemini cited ABN AMRO pages in my earlier research.

How is that possible? The firewall may refuse requests that only claim to be an AI bot and let the real bots in. The platform may have cited the page from a search index rather than fetching it live. Or the page was fetched before the rule existed. A test like mine can show that a block is likely, not that it is certain. Your server logs show what the real bots received.

My advice: Test your important pages as AI bots, and check your firewall or bot protection settings for rules about AI crawlers. Compare that with your robots.txt: if robots.txt says yes and the server says no, one of them is not doing what you intended.

What about JavaScript?

Because the major AI crawlers don't run JavaScript, websites that build their content in the browser can look empty to them. In this sample, none of the homepages I could read had that problem: all of them had their main text in the HTML.

That doesn't mean it never happens. I only tested homepages, and product filters, prices, reviews or tabs further into a website are more often loaded with JavaScript. If your website is built as a single-page application, check the pages that matter most.

Do you need llms.txt?

Agent readiness scanners give most of their points for new standards. In my sample, these were rare:

• llms.txt, a summary file for language models: 26% of the cited websites and 7% of the random ones.
• Content Signals, a robots.txt line that says whether AI may use your content for search, answers and training: 3 cited and 7 random websites. All seven random ones used it to say no to training.
• A Markdown version, a plain-text copy of the page for agents that ask for it: 3 cited and 1 random website.
• Link headers that point agents to machine-readable resources: none.

The llms.txt difference stands out, but it doesn't show that llms.txt helps. The cited websites are mostly larger companies with more technical teams, and they were cited before I checked. Google says you don't need new machine-readable files for its AI features. I haven't seen evidence that any of the major AI crawlers use llms.txt to choose sources.

Content Signals are a stated preference, not a block, and no AI company has confirmed that it follows them. Markdown versions are requested by some AI tools, such as Claude Code. That can help agents that do tasks, but no effect on AI citations has been shown.

What did I change on my own website?

Cloudflare has a scanner, isitagentready.com, that gives websites a level based on these new standards. My site came out at level 1. I added three things:

• A Content Signals line in robots.txt, allowing search, AI answers and training.
• A Markdown version of every page, served to agents that ask for it.
• A Link header that points agents to my sitemap.

That moved the site to level 3. Level 4 and higher asks for things like an MCP server or an agent card, which only make sense if you offer a service agents can use. I skipped those.

Did the changes help? To be honest: I don't know yet. They were quick to add and harmless, so I see them as a small experiment. I log which tools request the Markdown version and will report on that later. What I did not do is treat the higher level as proof that AI can use my site better. The checks that decide that, access and readable HTML, were already in order.

Which checks can you ignore?

Some scanners also mark websites down for standards that only apply to certain businesses:

• Payment protocols such as ACP, UCP and x402: for webshops that want AI agents to buy on a customer's behalf.
• MCP server cards, A2A agent cards, OAuth discovery and API catalogs: for websites with an API or a service that agents log into or call.
• WebMCP: lets a page offer actions to AI agents in the browser. Still experimental.

If none of these describe your business, a low score on them says nothing about whether ChatGPT or Claude can use your website.

Checklist: test your own website

1. Test your most important pages as AI bots. Use the AI Agent Readiness Checker, which compares a browser request with eight AI bots and robots.txt.
2. Read your robots.txt. Make sure the answer bots aren't blocked by accident, and that any training block names the training bots specifically.
3. Check your firewall and bot protection. Look for settings about AI crawlers or AI bots in your CDN, hosting panel or security plugin.
4. Look at your server logs. Search for OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot, and check which status codes they received.
5. Check that the content is in the HTML. View the page source, or see what the checker shows without JavaScript.

Being reachable is the first condition, not a guarantee. In my research into which pages AI cites, how well a page answered the question mattered most.

What can't this research tell you?

The samples are small: 147 and 125 websites. Treat the percentages as indications, not precise figures for all Dutch websites.

I tested homepages only, at one moment in November 2026. The checker sends requests with each bot's published user agent from a regular server; it can't send requests from OpenAI's, Anthropic's or Perplexity's own addresses. Firewalls that check those addresses may treat real bots differently, in both directions.

The cited websites are a selection of domains AI already cites. They show what successful websites do, not what causes citations. My first random sample, drawn from the full Tranco top million, contained a network of near-identical spam sites. I replaced it with a sample from the top 300,000.