# our website is readable by agents

What we built so AI agents can read spookat.com and act on it: llms.txt, markdown for every page, auth.md, OpenAPI, an MCP server card, WebMCP and an open robots.txt.

2026-10-16 · Kuba

Some of the people looking for a chat widget will never read our homepage. Their agent will. It gets asked "add live chat to my site", fetches a few URLs, and either finds what it needs or moves on to the next tool. So spookat.com is built to be read by agents as much as by people, and everything below is live and checkable with `curl`.

## llms.txt: the whole manual in one file

[`/llms.txt`](/llms.txt) is the entire product manual as plain markdown: install, config, Slack and Discord, webhooks, the reply API, MCP, partners. On 2026-09-27 it was 28.8 kb. It also lists every published blog post in English with a link to its markdown version.

The `/docs` pages are translated into 12 languages for humans. `llms.txt` stays in English, and so do the code blocks, prompts and tool names inside the translated docs, because that's what agents copy and run. A test fails the build if a translation touches any of them.

## every page as markdown

Ask any page on spookat.com for markdown and you get markdown:

```
curl -H "Accept: text/markdown" https://spookat.com/pricing/
```

That returns the page's main content as clean markdown, without the nav, the footer or the buttons. The build writes a markdown twin of every page, and that's what you get when the request asks for it. Blog posts also have their own `.md` URL, which is what `llms.txt` links to.

The homepage also answers every request with a `Link` header pointing at `/llms.txt` (`rel="describedby"`) and `/docs/` (`rel="service-doc"`), so an agent that only looks at headers still finds the manual.

## auth.md: how an agent gets in

[`/auth.md`](/auth.md) is a page written for agents, not for people. It explains the three ways in:

1. An MCP client with a human at a browser signs in with OAuth. The human picks the sites and a scope.
2. A human who already has an account hands the agent a key.
3. No account yet: the agent registers the site itself, gets the snippet to install and a claim link for its human.

The third one is the unusual part. An agent can set up Spookat on a site end to end, then its human clicks one link to claim it. Until someone claims it, the widget says nobody's answering yet. And whatever an agent does, a human always pays: an agent can start a plan change, and all it gets back is a checkout link for a person.

## openapi and an api catalog

The reply API is described in [`/openapi.json`](/openapi.json), OpenAPI 3.1, kept in step with the server by a test. [`/.well-known/api-catalog`](/.well-known/api-catalog) follows RFC 9727 and lists the API, its spec, the docs and the health check in one small JSON file. An agent that knows the standard finds everything from one well-known URL.

## an MCP server card

Spookat has a remote MCP server at `https://api.spookat.com/mcp`. Agents connect to it to read chats and reply, leave internal notes, hand a chat to a human, change the widget's look and connect Slack or Discord. [`/.well-known/mcp/server-card.json`](/.well-known/mcp/server-card.json) describes it: name, transport, where to authenticate, where the manual is. Registries and scanners that look for MCP servers find it without anyone filling in a form.

Adding it to Claude Code is one line:

```
claude mcp add --transport http spookat https://api.spookat.com/mcp
```

## WebMCP: the site itself has tools

Browsers are starting to let pages hand tools to the agent running in them. On spookat.com, when the browser supports it, the page registers tools to read the site and to set a chat up:

- `search_docs`: search the manual and get the matching sections back as markdown
- `read_page`: read any spookat.com page as markdown
- `open_page`: show the human a page, in the language they're reading the site in
- `join_waitlist`: put the human on the waitlist, only with their ok, on pages that have the waitlist form
- `register_site`: register the human's domain, no account needed, and get the install snippet and a claim link
- the setup tools (`set_style`, `set_accent`, `publish` and the rest), the same as on the MCP server, minus connecting a channel and paying
- `claim_site`: hand the site to the human, who claims it with one click

When the browser doesn't support it, nothing happens and nothing loads.

## robots.txt says yes

Our [`/robots.txt`](/robots.txt) invites crawlers in, and it says so in the first line. Search engines, AI assistants and model trainers are all welcome. Besides the default group, it names 23 AI crawlers one by one (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot, CCBot and others), so a stricter default somewhere upstream can't read silence as a no. The only path that's off limits is `/u/`, our analytics proxy, which has nothing to read.

Our CDN has its own switches for blocking AI crawlers. We keep every one of them off, including the one that rewrites your robots.txt for you.

## why this is worth doing

Because "add live chat to my site" is increasingly a sentence typed into an agent, not a search box. If the agent can read our manual in one request, install the widget from a snippet, and verify it without a human clicking through a dashboard, it will pick the tool that let it finish the job.

It's also just a better website. The markdown twins are the fastest way to read our docs, humans included, and a manual that fits in one file forces you to keep it short.

We're proud of this one. Most SaaS sites are a wall of JavaScript to an agent. Ours hands it the manual, the API spec, the MCP server and a way to sign itself up, all from URLs you can check right now.

Point your agent at [spookat.com/llms.txt](/llms.txt) and ask it to add a chat to your site. Then [start the trial](/pricing/) it will send you to.

---
source: https://spookat.com/blog/agent-readable-website/