---
name: harness-engineer
description: Apply when designing or reviewing any LLM harness — a system where an LLM makes decisions and calls tools inside deterministic scaffolding. Covers correctness vs policy separation, verifier-as-truth pattern, trust boundary for LLM-declared values, credential isolation, two-phase writes, structural correction over prompt directives, read/write authority split, single reasoning pass, and the anti-patterns that quietly ship. Portable across projects; not tied to a specific LLM, browser, or task domain. Triggers on phrases like "build an agent", "LLM harness", "LLM tool calling", "agent architecture", "AI safety scaffolding", "guardrails for LLM", "prompt injection", "should this be in the prompt or the code".
---

# harness-engineer

## Purpose

Guide the design and review of **LLM harnesses** — systems where an LLM makes decisions and calls tools inside deterministic scaffolding that bounds cost, blast radius, and correctness. Codifies eight rules distilled from building a production harness end-to-end, so future harness projects don't re-derive them by hitting the same failures.

## When to use

Apply automatically when the task involves any of:

- Designing an LLM that will call tools, drive a browser, or take real-world actions
- Reviewing an existing agent / LLM-tooling architecture
- Deciding whether a specific behavior belongs in the system prompt or in code
- Adding a guardrail, verifier, or retry policy to an LLM-driven flow
- Debugging non-deterministic failures in an LLM-driven system
- Handling credentials, PII, or destructive operations from an LLM

If the user is doing plain single-turn LLM calls with no tool use or side effects, this skill mostly doesn't apply.

## The mental model

Everything in this skill follows one principle:

> The LLM is a talented but chronically unreliable junior. It'll do the work. It may also lie about how it went. The harness's job is to check — and to make sure the LLM can't get into trouble even if it wants to.

Concretely:

- **The LLM is a worker.** It reasons, picks tools, produces text.
- **The harness is the supervisor.** It defines what tools exist, validates every call, verifies every claim of success, caps how much can be spent before giving up.
- **The user talks to the harness, not the LLM.** The harness mediates.

Everything below is a specific instance of "make the LLM be a worker, not a supervisor".

## The eight rules

### Rule 1 — The verifier, not the LLM's message, is the source of truth

The LLM's final message says whatever it feels like saying. Confident tone ≠ correct outcome. Never treat "the LLM said it succeeded" as evidence it succeeded.

**Pattern:** after every task attempt, run a deterministic **verifier** that reads the actual system state (URL, DOM, API response, DB row) and returns a structured verdict. That verdict decides the exit code, not the LLM's narration.

Display the LLM's message if you want, but label it something like `UNVERIFIED — may be wrong` and always show the verifier's verdict beside it.

**Red flag:** the exit code of your process depends on parsing the LLM's response text. Rewrite to depend on state you observed independently.

### Rule 2 — No prompt cooperation for correctness

Prompts are hints, not enforcement. If your correctness guarantee is "the LLM will follow this instruction", you have no guarantee — you have a hope.

**Test:** for every line in your system prompt, ask: *if the LLM ignores or misinterprets this line, does the system still behave correctly?*

- Yes → the line is fine (soft policy: response style, language, refusal scope).
- No → the correctness needs to move into code (guardrail, verifier, structural rejection).

The prompt is where you describe the tools (capability manifest) and set response style. Correctness — retry logic, verification, credential handling, target matching, action ordering — belongs in code where you can test it.

**Red flag:** you find yourself adding "please only do X when Y" to the prompt to fix a bug. That belongs in a guardrail.

### Rule 3 — Trust boundary: validate LLM-declared values against outside anchors

If you ask the LLM to structure a piece of input (extract a title, classify an intent, disambiguate a reference), **validate the LLM's output against something the LLM couldn't manipulate.**

The failure mode this catches: the LLM extracts value X and then satisfies a guardrail keyed on X. Circular.

**Pattern:** the LLM's declared value must match some invariant grounded outside the LLM's control. Common anchors:

- Substring of the user's original message
- Element present in an authoritative list (e.g. an item found by an unrelated lookup)
- Signature / checksum / provenance token generated before the LLM saw the input

If the declared value doesn't satisfy the anchor, refuse the declaration and let the LLM retry.

**Red flag:** the LLM declares a value that becomes the constraint on its own next action. Add an anchor.

### Rule 4 — Credentials, secrets, and PII never enter the LLM's world

The moment a form-fill / API-token / DB-write tool exists in the LLM's schema, three things become possible:

1. Prompt injection can redirect where the secret goes.
2. The LLM's context (and any logging of that context) now contains the secret.
3. Every future refactor has to remember not to log tool arguments.

**Pattern:** perform sensitive operations in Python code, not through LLM tool calls.

- Login? The harness fills the form directly. `fill()` primitive exists on the browser adapter but is deliberately not in the LLM's tool schema.
- API tokens? Read from env in the harness's HTTP client, never surfaced.
- User PII? Redact at the boundary. The LLM sees `{user_id: 42}`, not `{name: "…", ssn: "…"}`.

If a task needs the LLM to authenticate, the harness detects the auth-required state, pauses the LLM, does the auth in code, and resumes. The LLM never learns the secret existed.

**Red flag:** an LLM tool takes a `password`, `token`, `api_key`, or `credential` argument. Remove it.

### Rule 5 — Two-phase writes: pending → verified

When the LLM's actions have side effects (DB writes, external API calls, purchases), you'll want the LLM's next tool call to see the effect. But you also don't want to lock in an outcome that later turns out to be a false positive.

**Pattern:** two-phase writes.

- **Phase 1 (mid-run):** on evidence-of-success, insert the row / call the API / etc., but mark it "pending" (`verified=NULL`, `status=provisional`). Follow-up queries with `include_pending=True` see it — the LLM sees a truthful post-action count.
- **Phase 2 (attempt end):** after the verifier decides success/failure, update the pending row's status: `verified=True` on success, `verified=False` on failure. Default read paths only count `verified=True`.

This gives you three things at once: (1) LLM sees the fresh value mid-run, (2) audit trail records the false positives instead of silently discarding them, (3) durable state matches the verifier's decision, not the LLM's guess.

**Red flag:** you're either "write eagerly and hope" (broken audit trail) or "wait until fully verified" (LLM sees stale counts). Two-phase is the compromise.

### Rule 6 — Structural correction beats prompt directives

When the LLM might misbehave, the answer is almost never *"add a stronger instruction to the prompt"*. It's *"make the misbehavior impossible or self-correcting via a refused tool call"*.

**Pattern:** where you'd add "please do X first", instead:

- Add a required tool that must be called before the guarded action
- The pre-dispatch guardrail refuses the guarded action until the prerequisite tool has been called
- The rejection message tells the LLM exactly what to do next

Now if the LLM forgets, the rejection *is* the correction. It doesn't rely on the LLM having read the prompt line at all. The system prompt describing the tool becomes a nice-to-have hint, not a load-bearing instruction.

Bonus: structural correction is self-documenting. A new engineer reading the code sees the rejection reason and understands the invariant immediately.

**Red flag:** every bugfix adds another sentence to the system prompt. Rewrite the sentence as a refused tool call.

### Rule 7 — Read vs write authority split

Reads are LLM-callable. Writes are harness side effects. Always.

**Pattern:** the LLM's tool schema exposes only read actions (queries, page navigation, list lookups, count fetches). Writes happen in harness code, triggered by *observing* the results of the LLM's reads.

Concrete example: the LLM calls `click(upvote_arrow)`. The **read** side of that is the click's result. The **write** side (recording that the user just upvoted) is a harness reaction — post-tool hook watches the result, decides "yes, the state changed to voted", and writes the row itself. The LLM never has a `record_upvote` tool.

Enforce this at three layers:

1. **Schema** — write actions aren't in the tool schema.
2. **Pre-dispatch guardrail** — even if a malformed call slips through, an allowlist check refuses non-read actions.
3. **Dispatcher** — the dispatcher only routes read actions to their handlers.

Defence-in-depth. A compromised LLM has to break through three unrelated layers to write.

**Red flag:** the LLM has a tool named `create_*`, `update_*`, `delete_*`, `record_*`, `write_*`, `insert_*`, `mark_*`. Move it to a harness reaction to a *read* the LLM already does.

### Rule 8 — One reasoning pass, maximum output

Don't make the LLM reason twice about the same input. If you're tempted to add a separate LLM call to "pre-extract" or "pre-classify" something from the user's message, ask: *can the main loop's LLM do this reasoning as one of its tool calls?*

**Pattern:** expose the extraction as a first-class tool (`set_target`, `classify_intent`, `select_option`) and have the main-loop LLM call it. The rest of the flow keys off the value the LLM declared through the tool. The reasoning happens ONCE, in the same context that decides everything else.

Benefits:

- Fewer LLM calls (cost)
- One place where the reasoning lives (simpler debugging)
- Natural language phrasings automatically handled (the LLM did the parsing)
- Fewer moving parts (no separate extractor module, no regex fast-path, no fallback logic)

Combined with Rule 3 (anchor validation) and Rule 6 (structural rejection), this becomes: expose the tool, validate the declared value, refuse actions until the tool has been called. One reasoning pass, self-correcting, cost-optimal.

**Red flag:** your harness makes 2+ LLM calls per user request, and the second one is reasoning about the user's message the first one also saw. Consolidate.

## The one-line decision framework

Every time you're deciding where a behavior belongs, apply this:

> **Is this a guarantee I need to hold even when the model behaves badly?**
> - Yes → it's code (guardrail, verifier, structural rejection, adapter boundary).
> - No → it's prompt (response style, language, refusal scope, capability description).

Everything in this skill is a specific application of that question.

## Architecture patterns that generalize

### The single-yellow-node diagram

Draw your architecture. Color yellow every place that makes an LLM call. **Aim for exactly one yellow node.** Multiple yellow nodes means you're paying twice for reasoning that could happen once.

### Adapter boundary pattern

Every third-party API call (LLM SDK, browser driver, DB client, HTTP library) is wrapped in a try/except at the adapter boundary. Exceptions convert to `{ok: false, error: ...}` result dicts. Callers of the adapter never see the third-party exception types.

This lets you (a) test your business logic against a lightweight fake without pulling the third-party library into the test-time dependency graph, and (b) swap the underlying library without touching consumers.

Reserve `except Exception` for exactly three named boundaries: adapter methods, per-attempt catches in the harness (so one crashed attempt doesn't kill the run), and the process-level last-line-of-defense in `main()`.

### Post-tool hook mechanism

Give your agent loop an optional post-tool callback fired after every real tool result. The harness registers a closure that reacts to specific tool outcomes (this is Rule 7 in mechanism form: reads happen via the LLM's tool, writes happen in the hook watching those reads).

Adding a new reaction: extend the hook. Don't add task-specific branches to the agent loop.

### Pre-dispatch guardrail phase

Before every tool call, iterate a small ordered list of guardrail checks. Each check is a pure function returning either "pass" or a structured rejection. First rejection short-circuits the dispatch and the LLM sees the reason in its tool-result stream (Rule 6).

Guardrails must be:

- **Pure functions** — no LLM calls inside a guardrail (Rule 8).
- **Independent** — one guardrail failing shouldn't cascade into another.
- **Named** — every guardrail has a stable id used in logs and rejection reasons.

### Cost & telemetry as first-class

Every LLM call's usage flows into a running total. Print the total prominently at run end. If a new call site doesn't accumulate its usage, that's a bug — you'll silently miss cost regressions.

Same for correlation IDs: every log line for a single run carries the same ID so you can grep out one invocation from a shared log.

### The Protocol / interface substitution pattern

Anywhere the harness reads from a third-party library, expose a small structural interface (Python Protocol, TypeScript interface, Go interface, etc.) with just the methods the harness actually uses. Consumers depend on the interface, not the concrete class. Tests can substitute a fake implementing the interface without importing the third-party library at all.

## Anti-patterns to STOP on

If you catch yourself doing any of these, back out:

- **Prompt cooperation for correctness.** "Please do X" in the prompt as your safety net. Move it to a guardrail.
- **Circular trust.** LLM declares its own constraint. Add an anchor (Rule 3).
- **Mocking the layer under test.** Testing the verifier by mocking the browser proves nothing about the verifier's logic. Fake at the Protocol boundary instead.
- **Bundling refactor with bugfix.** "While I'm here, I'll also fix this other thing." Two commits, always.
- **Speculative abstraction.** Adding a Protocol / config knob "just in case". Add it when the concrete need appears, not before.
- **Silent failures.** A DB write that returns success on error is a lie. Log at least; usually re-raise or return a failure result.
- **Task-specific logic in the orchestrator.** Domain-specific code in a `Harness` / `Agent` / `Orchestrator` class is a leak. Move it to a task module the harness delegates to.
- **Magic constants scattered across modules.** Timeouts, retry caps, cost caps → one config file. Task-specific data (selectors, prompts, failure signals) → live with the task.
- **Multi-paragraph comments explaining WHAT the code does.** The code shows what. Comments explain the WHY — the constraint being encoded, the invariant being upheld.
- **A pull request that adds a tool named `record_*` or `write_*` or `create_*` to the LLM's schema.** Rewrite as a harness reaction (Rule 7).

## When in doubt

- **Draw the yellow-node diagram** first. If more than one yellow node, consolidate before proceeding.
- **List every prompt line** and apply the Rule 2 test. Move failed lines to code.
- **List every LLM-callable tool** and apply the Rule 7 test. Move writes to harness reactions.
- **Ask what invariant a specific behavior encodes** before choosing where it lives. The invariant tells you the layer.

## Further reading

For a longer-form narrative version of these rules with concrete failure examples, see the companion blog post *What I Learned Building an LLM Harness*. This skill is the actionable distillation; the blog is the "why does this rule exist" story.