We pointed our own QA agent at our own site

Cairn is our open-source autonomous QA agent: it walks a web app, grounds every locator against the live DOM, designs methodical test cases, generates self-repairing Playwright specs, and judges the result. So we did the obvious thing — we pointed it at plune.ai.

The run

Two cairn explore runs against the deployed site, steered by a short checklist of intent-level targets (nav, the funnel, the blog, the signup flow):

target	cases	green	Pilot verdict	grounding	locator quality
plune.ai	11	55%	needs-work	1.00	1.00
plune.ai/cairn	13	62%	fail	1.00	1.00

grounding 1.00 / locator_quality 1.00 mean every locator Cairn wrote resolved to a real, user-facing element — no hallucinated selectors. The low green % is the interesting part: why did a third of the tests fail?

What it caught — a real bug, three layers deep

The headline finding: our product-updates signup was silently broken in production — and chasing Cairn’s failing “valid email → success” case peeled back three prod-only config gaps, each masking the next, none visible to CI, a build, or local dev:

1. Missing env var → 503 "Signups are not configured yet." The API needs three runtime vars (RESEND_API_KEY, RESEND_AUDIENCE_ID, SUBSCRIBE_SECRET), but docker-compose.yml forwarded only two — SUBSCRIBE_SECRET never reached the container.
2. A too-narrow key → 502. The Resend key had Sending access only, but the double-opt-in flow’s first step adds the address to an audience (contacts.create), which needs full access (401 restricted_api_key).
3. Wrong audience id → 502. RESEND_AUDIENCE_ID pointed at an audience that didn’t exist in the key’s account (404 Audience not found).

Each lived only in the server’s .env / Resend account — the one place CI and astro check can’t see; local dev worked the whole time, so nothing ever turned red. This is exactly what a dogfooding agent against production is for. (We also added server-side logging of the real Resend error, so the next such chain is a ten-second look at the logs, not a ten-step hunt.)

What it got wrong — the honest part

A QA agent isn’t an oracle, and a dogfooding post that pretends otherwise isn’t worth reading. Of the five failures on the homepage, only the signup one was a real defect. The rest were the agent over-asserting:

• Native validation. Our email field uses the browser’s built-in type=email / required validation — a native bubble, not on-page text. Cairn asserted getByText('Email is required') and friends, which don’t exist, and marked them failed. The site is correct; the assertions weren’t.
• A rule we never made. It expected a “must be less than 255 characters” message. We have no such limit.
• Duplicate labels. The eval-action link appears in both the body and the footer; a locator that didn’t disambiguate tripped Playwright’s strict mode. Two links, not zero.

Net: 1 real defect (three config root causes), 4 false positives — and triaging that split is still a human job.

The metrics, explained

Every run scores itself — green %, grounding, locator quality, redundancy, plus LLM-judge scores and one holistic Pilot verdict (pass / needs-work / fail). What each means: Metrics.

Try it on your own app

npm i -g @plune-ai/cairn
cairn explore --url https://your-app.example.com

Docs: getting started · GitHub.