Robots.txt in the AI era, should you block AI crawlers?

For years, robots.txt was a fairly settled topic: a small text file that told search engine bots which parts of your site to crawl and which to leave alone. The rise of AI systems that crawl the web to train models or answer questions in real time has added a new layer of decisions on top of that. Now the question isn’t just “should Googlebot see this page,” it’s also “should an AI company’s crawler be allowed to copy my content for training data, or to answer questions using my content without sending a visitor to my site at all.”

This guide covers what robots.txt actually does and doesn’t control, how AI crawlers differ from traditional search bots, and how to think through the decision of whether to allow or block them.

What robots.txt actually controls

Robots.txt is a plain text file placed at the root of your domain that gives instructions to well-behaved crawlers about which parts of your site they’re allowed to access. It works on an honor system: it’s a request, not a technical barrier, so any crawler that chooses to ignore it can still access your content unless you block it through other means, such as server-level rules.

A basic robots.txt file allows or disallows access by user agent, meaning you can write different rules for different bots. This is exactly why AI crawlers introduced a new wrinkle: each major AI company generally has its own identifiable crawler, which means you can, in principle, allow search engine crawlers while blocking specific AI training crawlers, or vice versa, rather than treating “block everything” or “allow everything” as your only two options.

How AI crawlers differ from search bots

Traditional search engine crawlers exist to index your content so it can appear in search results, which in turn sends visitors to your site when they click through. The incentive is roughly aligned: you want to be found, and the crawler wants to help people find you.

AI crawlers serve a few different purposes, and they don’t all carry the same trade-off:

• Training crawlers collect content to train AI models, generally without sending any direct traffic back to your site in return, now or later.
• Retrieval or “answer engine” crawlers fetch content in real time to answer a specific user question, sometimes citing and linking back to the source, sometimes not, depending on the product.
• Search-adjacent AI crawlers, used by AI features built into existing search engines, often power AI-generated summaries that appear above traditional search results, which can reduce click-through to the original source even when it’s cited.

Because these serve different purposes with different (or absent) value exchanges, treating “AI crawlers” as one single category to allow or block is often the wrong framing. It’s worth deciding per purpose, and where possible per crawler, rather than as a blanket policy.

Deciding whether to allow or block

There isn’t a universally correct answer here, it depends on what you’re trying to protect and what you’re trying to gain:

• If visibility in AI-generated answers matters to your traffic strategy, blocking the crawlers behind those answer engines could reduce your presence in a channel that’s becoming more relevant for some types of search behavior.
• If you’re primarily concerned about your content being used to train models without attribution or compensation, blocking specific training crawlers addresses that concern directly, though it doesn’t retroactively remove content already collected.
• If your content is your core commercial asset, such as original research, journalism, or paid content previews, the calculation leans more toward restriction, since the value being extracted is closer to your core business.
• If your goal is broad discoverability, such as for a small business site, allowing most crawlers, including AI ones, may support that goal better than blocking them.

There’s no single right choice, and many sites end up allowing some AI crawlers while blocking others, based on which value exchange they’re comfortable with.

Writing rules for specific crawlers

Robots.txt rules are written per user agent, so you can write a block for one crawler without affecting others. A simple example structure looks like:

User-agent: SomeAICrawler
Disallow: /

User-agent: *
Allow: /

This blocks one named crawler entirely while leaving the default rule open for everyone else. The specific user agent names change over time as AI companies launch and rename crawlers, so it’s worth checking current, published crawler names rather than relying on an old list, since a rule written for an outdated user agent name won’t match the crawler actually visiting your site.

Generating a robots.txt file

Getting the syntax exactly right matters, since a small mistake, like a misplaced slash or an incorrectly scoped rule, can accidentally block far more (or less) than intended, including blocking search engines entirely by mistake. Holsha’s free robots.txt generator helps you build a correctly formatted file by specifying which crawlers to allow or block and which paths to include, reducing the risk of a syntax error that silently breaks your intended rules.

Robots.txt is not a security tool

It’s worth repeating that robots.txt is a voluntary instruction, not an access control mechanism. Any crawler, AI-related or not, that doesn’t respect the standard can simply ignore it. If you need to genuinely prevent access to specific content, that requires authentication, server-level blocking, or another technical control, not just a robots.txt rule. Robots.txt is best understood as a signal of intent to well-behaved crawlers, not a wall.

Robots.txt as part of a bigger SEO setup

Managing crawler access is one part of a broader technical SEO setup. Once you’ve decided which crawlers to allow, it’s worth making sure the pages you do want indexed are properly described with structured data, which is covered in schema markup for beginners, and that their titles and descriptions are written to earn the click once they do appear in results, covered in how to write a meta title and description that get clicks.

FAQ

Does blocking AI crawlers hurt my search engine ranking? Blocking an AI-specific crawler generally doesn’t affect traditional search rankings, since search indexing typically uses a separate crawler with its own user agent, which you can leave unaffected.

Can I block AI crawlers but still allow Google and Bing? Yes, this is one of the most common configurations, since robots.txt rules are written per user agent, allowing you to write separate rules for each crawler.

Will an AI company definitely respect my robots.txt rule? Reputable AI companies generally state that they respect robots.txt, but the standard is voluntary, so there’s no technical guarantee for every crawler that might visit your site.

Do I need to update my robots.txt file regularly? It’s worth reviewing periodically, since new crawlers appear and user agent names for existing ones can change, meaning a rule that worked previously may stop matching the crawler it was meant to target.

Summary

Robots.txt gives you a way to state your preferences to crawlers, including the growing list of AI-related ones, but the right configuration depends on what you’re trying to protect and what you’d gain from allowing access. Decide per crawler rather than treating “AI” as one category, and use Holsha’s free robots.txt generator to make sure the rules you write actually do what you intend.