What we found

57,740 sites invite agents in. About half have built the door.

We read every robots.txt in a full public crawl of the web — 83.9 million of them — looking for sites that tell AI agents they're welcome. Two million carry a declared stance on AI use; tens of thousands explicitly invite agents in. When we probed those sites live, roughly half had actually built something an agent can use.

Known-Good exists to close that gap: to find the parts of the web an agent can genuinely use, and prove it by testing rather than trusting the label.

83.9M

robots.txt files read, across a full public crawl of the web

2,171,626

of them carry a content signal — a declared stance on AI use

57,740

domains that explicitly welcome AI agents to read them

~1 in 2

of those that decided for themselves have built something usable

Source: a full public crawl of the web, 2026, parsed group-aware. The ~1-in-2 figure is from 1,500 live probes; full verification is running now.

Machine representation

The same page, without the page.

This is the substance of this site as an agent would take it — no layout, no navigation. The full twin is at /index.md.

# Known-Good

mission:  find the parts of the web an agent can genuinely use, and prove it
status:   crawling; index verification in progress
version:  0.1
contact:  hello@knowngood.sh

## Findings

robots_txt_read:        83900812
carrying_content_signal: 2171626
domains_inviting_agents: 57740
implemented_rate:       ~0.5   # of self-declared sites, n=1500 live probes
cdn_default_share:      0.86   # declarations that are one CDN's default

## To make a site agent-ready

1_declare:   Content-Signal: ai-input=yes   # in /robots.txt
2_serve:     a markdown twin, or text/markdown on Accept negotiation
3_expose:    /llms.txt, /.well-known/api-catalog, /.well-known/mcp/server-card.json

## Live today

/llms.txt       curated index of this site, for language models
/index.md       this page, as markdown
/bot            crawler identity, opt-out, published IPs
/bot/ips.json   verify a request is genuinely ours
/robots.txt     our own content signals: ai-input=yes, ai-train=no

This switch is CSS only — no JavaScript ran to render either view.

Making a site agent-ready

Three things, in order of effort. The first takes a minute and costs nothing. Some call the second and third going headless — serving what your site can do without requiring anyone, or anything, to read the page.

Say agents are welcome

One line in robots.txt tells every AI crawler what you permit. It is a consent signal, not a capability — but nothing else counts until it's there.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no

ai-input=yes → agents may use your content to answer someone's question. ai-train=no → but not to train models. You choose.

Offer a machine-readable version

Your pages are built for eyes: navigation, banners, scripts. An agent has to fight through all of it. Serve the same content as clean markdown — either at a predictable URL, or by answering the Accept header.

GET /pricing        Accept: text/html      → your page
GET /pricing        Accept: text/markdown  → clean markdown
GET /pricing.md                            → same, predictable URL

This is the single strongest signal we measure, and the rarest.

Publish endpoints an agent can act on

Reading is half of it. If an agent can book, buy, query or submit on your site, say so somewhere it can find without guessing.

/llms.txt                            what you offer, in plain text
/.well-known/api-catalog             your APIs (RFC 9727)
/.well-known/mcp/server-card.json    your MCP server, if you have one

A 200 that returns an HTML error page is worse than a clean 404 — it sends an agent down a path that breaks.