honest-coverage states plainly
what a green test run does and doesn’t prove — a Stable/Moderate/Blocked
rating per test, not a generic disclaimer. Its own opening frames the
problem it’s generalized from: “a reviewer who counts spec files concludes
there’s a large suite. A reviewer who opens one concludes the opposite.” A
green suite can quietly mean less than it appears to.
This skill is that instinct’s sibling, aimed at production instead of the test run. A green suite proves a code path behaves correctly right now, under test conditions. It says nothing about whether anyone would find out if that same path started failing for real users tomorrow. Those are independent properties — a well-tested feature can be completely unwatched in production, and a well-monitored one can be undertested. This skill checks the second property specifically: not “is this correct,” but “if this breaks, does anything downstream of the code know.”
Generalizes the Google SRE book’s ch. 6 (Monitoring Distributed Systems): the chapter’s core claim is that a system you cannot observe is a system you can only debug after a user files a report, and that the gap between “looks monitored” and “is monitored” is where outages hide the longest.
For each failure path checked:
Path: <function/file/flow and the specific failure being asked about>
Failure shape: <exception | wrong value | silent non-completion | ...>
On failure: <what actually happens right now — quote the code>
Verdict: <observed | silent | false-observed>
Gap: <the specific thing missing, named — not "add logging">
Where Verdict is one of:
| Verdict | Meaning |
|---|---|
| Observed | The failure path has a log line, metric, or error-tracking hook that actually fires and carries enough detail to diagnose from. |
| Silent | Nothing runs on this path at all — no log, no metric, no hook. The failure vanishes. |
| False-observed | Something that looks like observability sits on the path — a catch block, a log statement — but it doesn’t actually fire, or fires without the detail needed to diagnose anything. Worse than Silent, because it reads as covered on a code review. |
Distinguish “no observability” from “observability that never fires.” A
bare catch (e) {} or a catch (e) { return null; } is not more observable
than no try/catch at all — it’s less safe than that, because it looks
handled. Reviewing code for a catch block’s presence is not this skill;
reviewing what’s actually inside it is.
Name the failure path and the specific missing piece, never “add more
logging.” “This function could use better error handling” is not a
finding — it’s the thing this skill exists to replace. “The catch block
in pollRunStatus (useRun.ts) swallows the fetch error into a state field
the UI reads, but nothing writes it anywhere durable — a user who never
looks at that screen mid-poll never surfaces the failure, and no metric
increments” is.
A user-visible error message is not the same claim as an operator-visible one. A toast that says “Something went wrong” tells the one user who hit it. It doesn’t tell whoever’s on call that it’s happening to everyone, at what rate, or whether it just started. Check both separately — this skill’s job is the second one; a UI-facing error banner passing does not close this skill’s finding.
Ask what the log or metric would need to contain to actually diagnose from, not just whether one exists. A log line that fires but omits the input that caused the failure is closer to Silent than Observed — it tells you that something broke, not what.
Rank by how quietly a break could go unnoticed, not by how likely the break is. A rarely-hit path with zero observability that could fail for days before a user complains outranks a common path with the same gap that someone would notice within the hour just from support volume.
This is not reliability-audit. That skill asks whether a call to an
external dependency degrades gracefully — a timeout, a bounded retry, a
capped queue. This skill asks whether any failure on a path, degraded or
not, leaves a trace anyone would see. A path can have perfect timeout
handling and still be this skill’s finding, if the timeout itself is never
logged or counted anywhere.
Applied to write-playwright-test’s own worked example’s failure
handling (the login page’s invalid-credentials case, as implied by the
scenario description — no actual frontend code exists for it in this repo,
so this reasons from the described behavior, stated as such):
Path: Login form submission, invalid-credentials branch
Failure shape: Server returns an auth failure; the page shows an inline
error and stays on /login (per the scenario description).
On failure: Not knowable from the description alone — no server-side
code was provided alongside the scenario, only the observed
client behavior the test asserts against.
Verdict: Cannot be rated from a test scenario alone — this is the
limit of what this skill can check without the
implementation. Flagging that limit explicitly, rather than
guessing, is itself the honest-coverage discipline applied
to this skill's own output.
Gap: Ask for the server-side handler before rating this path —
specifically, whether a spike in invalid-credential
responses (which could mean a credential-stuffing attempt,
not just a typo) increments any counter, or whether it's
indistinguishable from background noise in whatever the app
currently logs.
A gap this skill is built to actually catch, from useRun.ts’s real
polling code (the same file refactor-audit’s trial in
CALIBRATION.md and reliability-audit’s worked example both already
examined, for different properties):
Path: pollRunStatus, src/app/_hooks/useRun.ts
Failure shape: fetch throws, or the response's !response.ok branch fires.
On failure: The catch block sets a typed fallback message into
component state (setRunStatus/setResultError-shaped), which
the UI can render if the user is looking at this screen at
that moment.
Verdict: False-observed. The failure is caught and given a
user-facing message, which reads as "handled" on review -
but nothing here writes to a log, increments a metric, or
reports to an error tracker. It is entirely dependent on a
human being looking at the right screen at the right
second.
Gap: No durable record of a poll failure exists once the
component unmounts or the user navigates away. A run that
starts failing in a tab nobody's watching produces zero
trace. Missing specifically: a console.error or logger call
in the catch block carrying the run ID and status code, at
minimum - not a general call to "add observability."
Release engineering (SRE ch. 8 — deployment, rollback, canarying) is
out of scope here on purpose, not an oversight. It’s already covered by the
existing verified-release-pipeline account skill; duplicating it under a
different name would just split one concern across two places.
“Run observability-check on this feature — if it breaks in prod, would we know?”
Or paired with its sibling: “run honest-coverage and observability-check on this together” — one rates what the tests prove, the other rates what production would tell you if they were wrong.