Robots.txt and AI training: one signal to say no (and the scribes' curses)

On 30 October OpenAI announced that it will honor a shared robots.txt signal that lets website operators state their AI training preferences to participating providers. If you run a company website, with catalogs, product pages and articles a model might read, it deserves five minutes. But be careful about what it promises and what it doesn't.

What happened, without embroidery

OpenAI's announcement, as summarized in its own feed, says the company will honor a shared robots.txt signal designed to let a site declare its preferences once. For technical details and timing, check the official page, which was not readable by every client when I looked: so I report nothing I could not verify.

The technical basis, on the other hand, is public. On the same day, 30 October, an individual draft appeared in the IETF registry, Machine-Readable Training Preference Signal, authored by Kevin Kelley of Anthropic. The document states that it is not endorsed by the IETF and has no formal standing. It expires on 3 May 2027.

How it would work

Robots.txt is the file where a site gives instructions to the programs that visit it, under the Robots Exclusion Protocol (RFC 9309). The draft invents no new rules: it uses a "User-Agent" group with a special name, __usage_ai-training. An example from the draft:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

It means: I prefer that no content be used to train an AI, except the /press-kit/ folder. A useful detail: generic User-Agent: * groups do not apply to this signal; you have to write it explicitly. The broader standardization work belongs to the IETF AI Preferences working group, which deals with vocabulary and ways to attach preferences to content.

What it does not guarantee

• It is not a standard. It is an individual draft, and the vocabulary may change.
• It is voluntary. RFC 9309 itself reminds us that these rules "are not a form of access authorization."
• It is not a crawl rule. To limit individual crawlers you still need their specific groups.
• Private agreements can override it, the draft says.
• It says nothing about data already collected: it is not a deletion.

The medieval case: the curse at the end of the book

And you'll say: what does the Middle Ages have to do with this? Quite a lot. When a book cost a scribe months of work, the way to protect it was to write, usually at the end, in the colophon, a curse: whoever steals this, let them be excommunicated, struck by anathema. Hundreds survive; the oldest cited in the collections comes from the monastery of St Gall, in 880, and another closes a Bible of 1172 now in the British Library. Marc Drogin gathered many of them in Anathema! (1983). Beware, though: the famous curse "of San Pedro in Barcelona" is a twentieth-century hoax.

The point is this: the curse was a statement of preference, written where anyone opening the book would read it. It worked on people who respected the Church, not on those who did not fear it. Well, your robots.txt today sits in that colophon: it says what you prefer, but it does not fit a lock. For truly confidential data you need protected access, not a label.