The expectation going in was that brands are quietly locking themselves out of AI answers. The data says something less dramatic and more useful: in commerce, there is barely a decision to find.

What we checked

One file per domain — /robots.txt — across 161 sites in three groups: D2C and consumer brands, software and SaaS companies, and a contrast group of publishers and media. 117 responded.

For each site we asked, per crawler, whether it is blocked deliberately (the site names the agent and disallows it), blocked by accident (the agent is not named but a User-agent: * group disallows everything), or allowed. The distinction matters more than the totals, and almost no summary of this topic draws it.

The publishers are not a market Elivate SEO sells to. They are in the sample because a number needs something to sit next to. “Almost nobody blocks AI crawlers” is an empty sentence until you can point at a sector where almost everybody does.

The domain list and the raw results are in the repository, so the run is reproducible rather than merely reported.

Commerce has no position

Of 105 D2C and SaaS sites that responded, 86 never mention an AI crawler at all.

Under the Robots Exclusion Protocol, anything not disallowed is allowed. So those 86 sites do permit every AI crawler — but by inheritance, not by choice. Their robots.txt was written before the question existed and nobody has been back to it.

Of the 19 that do name at least one AI agent, 16 explicitly allow. Only one site in the entire commerce sample blocks a crawler that fetches pages to answer a live question.

Among the D2C brands specifically, not one blocks anything. The four that name an AI agent — Yeti, Mejuri, Bôll & Branch and Summer Fridays — all name them in order to allow them.

Publishing has decided, unanimously

Of the 12 publishers that responded, 12 block AI crawlers. Not a majority. All of them.

And they are not only blocking training. All 12 block at least one retrieval crawler — the kind that fetches a page because someone asked a question a minute ago. The New York Times, Bloomberg and CNN block every agent we tested.

That is a coherent position for a business whose product is the archive itself. If your revenue comes from people paying to read your words, being summarised for free is the threat, and being cited is small compensation.

It is also the opposite of what a business selling something else should want, which is the whole point of putting the two groups side by side.

The distinction nearly everyone misses

“AI crawler” describes two jobs that have nothing to do with each other.

Training crawlers — GPTBot, ClaudeBot, Google-Extended, CCBot — collect text to train models. Blocking them is a content licensing decision. It costs a business nothing in visibility.

Retrieval crawlers — OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot — fetch a page because a user just asked something. Blocking these removes a site from the answer.

Almost every robots.txt we read treats the two as one category. One site in the sample did not: Warby Parker blocks CCBot and Bytespider, the bulk scrapers, and leaves the answer engines alone. That is the only file in 117 that reads like someone understood what they were choosing between.

What to actually do about it

If your robots.txt has never named an AI crawler, you have not made a decision — you have inherited one. It happens to be the right default for most businesses, which is a reason to confirm it, not a reason to ignore it.

Three steps, in order:

  1. Read your robots.txt. Most people selling something have never opened it.
  2. Decide about training separately from retrieval. If you don’t want your content in training corpora, block the training crawlers. That choice does not have to cost you a place in answers.
  3. Never block a retrieval crawler unless you mean it. If someone asks an assistant a question your page answers, this is the only thing standing between you and being the answer.

The cheapest mistake to avoid is copying a publisher’s robots.txt because it looks thorough. Their incentives are the reverse of yours.

What this does not show

The sample is a named set, not a random draw. These are 161 recognisable companies, skewed large and well-resourced, so this supports “of these companies, X do Y” and not “X% of D2C brands”. A small store on a stock theme may look entirely different, and that is a study we have not run.

The publisher group is the weakest part. Of 30 publishers, 17 refused this crawler at the network layer before robots.txt was ever reached — leaving 12. Those refusals are consistent with a sector that guards access aggressively, but they are not evidence for it, and treating them as such would be exactly the kind of thing this article is arguing against.

Everything above is one measurement on one day. The interesting version is the same run in six months, when some of those 86 silent sites have decided.