# Known-Good

> Finding the parts of the web an agent can genuinely use — and proving it by testing,
> rather than trusting the label.

- mission: find the parts of the web an agent can genuinely use, and prove it
- status: crawling; index verification in progress
- version: 0.1
- contact: hello@knowngood.sh

## What we found

We read every robots.txt in a full public crawl of the web — 83.9 million of them — looking
for sites that tell AI agents they're welcome. Two million carry a declared stance on AI use;
tens of thousands explicitly invite agents in. When we probed those sites live, roughly half had
actually built something an agent can use.

Source: a full public crawl of the web, 2026, parsed group-aware. The ~1-in-2 figure is from
1,500 live probes; full verification is running now.

- robots_txt_read: 83,900,812
- carrying_content_signal: 2,171,626
- domains_inviting_agents: 57,740
- implemented_rate: ~0.5 (of sites that decided for themselves; n=1,500 live probes)
- cdn_default_share: 0.86 (declarations that are one CDN's default, not a decision)

## Making a site agent-ready

Three things, in order of effort. The first takes a minute and costs nothing. Some call the
second and third going *headless* — serving what your site can do without requiring anyone,
or anything, to read the page.

### 1. Say agents are welcome

One line in `robots.txt` tells every AI crawler what you permit. It is a consent signal, not
a capability — but nothing else counts until it's there.

```
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
```

`ai-input=yes` means agents may use your content to answer someone's question.
`ai-train=no` means not to train models on it. You choose.

### 2. Offer a machine-readable version

Your pages are built for eyes: navigation, banners, scripts. An agent has to fight through
all of it. Serve the same content as clean markdown — either at a predictable URL, or by
answering the `Accept` header.

```
GET /pricing     Accept: text/html      -> your page
GET /pricing     Accept: text/markdown  -> clean markdown
GET /pricing.md                         -> same, predictable URL
```

This is the single strongest signal we measure, and the rarest.

### 3. Publish endpoints an agent can act on

Reading is half of it. If an agent can book, buy, query or submit on your site, say so
somewhere it can find without guessing.

```
/llms.txt                            what you offer, in plain text
/.well-known/api-catalog             your APIs (RFC 9727)
/.well-known/mcp/server-card.json    your MCP server, if you have one
```

A 200 that returns an HTML error page is worse than a clean 404 — it sends an agent down a
path that breaks.

## What's live here today

We grade other sites on this, so we publish our own state plainly. The rest — content
negotiation, the API, the MCP server — arrives with v1.

- `/index.md` — markdown twin of this page
- `/llms.txt` — curated index of this site, for language models
- `/robots.txt` — content signals, declared honestly: `ai-input=yes`, `ai-train=no`
- `/bot` and `/bot/ips.json` — crawler transparency, opt-out, published IPs

## Free agent-readiness review

Thirty minutes over video. We walk your site the way an agent does, show you where it gets
stuck, and what to fix first. No pitch, no obligation.

Book: https://cal.com/knowngood.sh/30min