What I learned building an LLM harness

I set out to build something trivial: a Python script that would let me ask "upvote the servo story on HN and tell me how many upvotes I've done" in plain English, and have an LLM drive a real browser to do it. First draft was ~80 lines. It worked once, felt magical, then quietly betrayed me in every conceivable way over the next few days.

• It claimed to have upvoted stories when it had actually landed on the login page.
• It upvoted the wrong story when the requested title had a typo.
• It burned 30 API calls in a loop trying variations of the same wrong action.
• It responded in Nepali because I typed a transliterated Nepali phrase.
• It cheerfully told me "you now have 6 upvotes" after doing absolutely nothing.

Every one of those failures pointed at the same underlying question: how much can you trust the LLM's own report of what it did? The answer, in every case, was "less than I did."

The thing that emerged from patching each of those failures is called a harness. This post is what I wish I'd known on day one.

What is an LLM harness?

A harness is the deterministic scaffolding around an LLM that makes the whole system bounded — bounded in cost, bounded in blast radius, bounded in the ways it can lie to you. Without a harness, you just have "an LLM with tool access", and that's a security bug, a cost bug, and a reliability bug wrapped in a demo.

Concretely, in a harness:

• The LLM is a worker, not a supervisor. It reasons, picks tools, and produces text.
• The harness is the supervisor. It defines what tools exist, validates every tool call, verifies every claim of success, and caps how much the LLM can spend before giving up.
• The user never talks to the LLM directly. They talk to the harness, which mediates.

The right analogy isn't "AI agent". It's a browser — a sandbox with a scripting engine inside, where the scripting engine can do anything the sandbox lets it do, and nothing else.

The mental model

Everything else in this post comes from one core mindset shift:

Treat the LLM as a talented but chronically unreliable junior. It'll do the work. It might also lie about how it went. Your job is to check.

The harness is where the checking lives.

Here's the architecture I ended up with, mapped to that idea:

Only one node is yellow. That's the only place LLM reasoning happens. Everything else — retry decisions, verification, DB writes, the final report — is Python that always does the same thing given the same inputs. That single-yellow-node discipline is where most harness bugs get fixed.

Rule 1 — The verifier, not the LLM, is the source of truth

The LLM's final message says whatever the LLM feels like saying. If the model was RLHF'd to be cheerful, the message will be cheerful. If it was trained to sound confident, the message will sound confident. Neither has any relationship to whether the task actually succeeded.

I saw this the moment I started running the tool for real. A run ending with:

"I upvoted story rank 3 successfully!"

Where "success" was the browser landing on https://news.ycombinator.com/vote?id=… — HN's "sorry, you have to be logged in" redirect page. The LLM had no way to know it had failed. It just narrated a plausible outcome.

The fix: run a verifier after every attempt. The verifier reads the browser state directly (URL, first 500 chars of body, "logged in" indicator on listing pages) and returns a structured verdict:

{"success": False, "reason": "landed on failure url (/vote?)", "terminal": True}

That verdict — not the LLM's message — decides the exit code. The LLM's message still gets displayed, but with a big label:

--- Last attempt: LLM claim (UNVERIFIED — may be wrong) ---
I upvoted story rank 3 successfully!

--- Last attempt: deterministic verifier said ---
FAIL: landed on failure url (/vote?)

The reader always sees both. The truth is always the second one.

Rule 2 — No prompt cooperation for correctness

Once you accept the LLM might lie, a second temptation appears: fix the lying with better prompts. "Please only claim success if you actually saw the upvote arrow disappear." "Please retry if you land on the login page." "Please do X, then Y, then Z."

Every one of these is a bug in disguise.

The rule I settled on: if the LLM ignores or misinterprets a line in the system prompt, does the harness still produce a correct outcome?

• Yes → the prompt line is fine (soft guidance, UX polish).
• No → you've built your correctness guarantee on quicksand.

Prompt is where you describe the tools (capability manifest) and set response style (language, refusal scope). Correctness — retry, verification, credential handling, target matching, ordering — lives in Python where you can test it.

The test: could you delete the entire system prompt and would the harness still refuse to do wrong things? If yes, you're safe. If no, refactor.

Rule 3 — Trust boundary: validate LLM-declared values against something outside the LLM's control

The most subtle bug I hit was in an early version of a "target-match guardrail". The idea was: if the user says "upvote 'The Loss of Bikash Mind'", only allow the LLM to click an upvote arrow whose story title equals "the loss of bikash mind". Simple.

Except: the LLM's job included telling me what the target was. So the LLM would extract the target as, say, "june in servo" (the story it was ABOUT to click), install that as the constraint, and then satisfy its own constraint by clicking exactly what it had already decided to click. Circular.

The fix: anything the LLM declares must be validated against the user's original message. Substring check, post-normalize. If the LLM declares a title that isn't in the user's message, the declaration is refused and the LLM has to try again.

That's the trust boundary in one sentence: the LLM does not get to invent the constraint it's supposed to be constrained by. It can only pick from the user's own words.

The same rule applies anywhere you want the LLM to structure user input — extraction, classification, disambiguation. Always anchor the LLM's output to something the LLM couldn't manipulate.

Rule 4 — Credentials never enter the LLM's world

There's a temptation, when the LLM needs to log in, to just… let it log in. Give it a fill_form(selector, value) tool. It's an LLM! It can figure out the fields.

Do not.

The moment a form-fill tool exists, three things become possible:

1. Prompt injection can redirect where the credentials go.
2. The LLM's context (and any downstream logging/trace) now contains the password.
3. Every future refactor has to remember not to accidentally log the tool arguments.

The pattern I use: the harness performs the login itself. The browser adapter exposes a fill() primitive, but it's deliberately absent from the LLM's tool schema. Only Python code — specifically the login handler — can call it. The password reads from an env var, gets typed straight into the form, and the LLM never sees the string.

If the LLM's session cookie expires mid-run, the harness detects it (the verifier's "logged in?" check catches it), pauses the attempt, runs the login handler, and resumes. From the LLM's perspective, one attempt just "failed" and the next attempt "worked". It has no idea a login happened.

Rule 5 — Two-phase writes: pending → verified

I wanted this behavior: user says "upvote a story and tell me the total". The LLM upvotes, then queries the count, and reports the fresh number.

Naive: LLM clicks, harness writes to DB, LLM queries count, count includes the new row.

Bug: what if the upvote didn't actually land? The DB row was written on the assumption of success. Now the count is wrong forever.

Solution: two-phase writes.

• Phase 1 (mid-run): when the harness sees evidence the upvote landed (already_voted=true in a follow-up get_stories), it inserts a DB row with verified=NULL. The count query returns it (using include_pending=True) so the LLM sees the truthful post-upvote number.
• Phase 2 (attempt end): after the verifier runs, the harness marks every pending row verified=True (if the verifier said success) or verified=False (if it said failure). The default count_upvotes only counts verified=True.

Result: the LLM's mid-run count is always correct. The audit trail is honest. False positives get flagged as verified=False for later investigation instead of being invisibly deleted.

Sequence diagram of the whole thing:

Rule 6 — Structural correction beats prompt directives

The single biggest architectural improvement I made was replacing a class of problems that looked like "please LLM, do this" with problems that look like "we refuse to run until you do this".

Concrete example. I had a target-match guardrail that only worked when the harness knew the user's requested title. Version 1: I wrote a helper LLM call at the start of every run: "read the user's message and return the target title as JSON." Two LLM calls per run instead of one, and a fragile regex-vs-LLM cascade to try to skip the extra call when possible.

Version 2 killed the extra call entirely. Instead, I added a tool called set_target(title) to the LLM's tool schema, and a rule to the pre-dispatch guardrail: any upvote click before set_target has been called is refused.

Now the LLM's own reasoning does the extraction, inside the same reasoning pass that decides everything else. And if the LLM forgets to call set_target, the rejection message tells it exactly what to do next:

The magic isn't the tool. The magic is that structural rejection is self-documenting. The LLM doesn't need to be told what to do — it gets told the exact reason the last thing failed, and figures it out.

Rule of thumb: whenever you're tempted to add an instruction to the prompt, ask if a refused tool call could deliver the same message more reliably. It usually can.

Rule 7 — Read vs write authority split

Reads are LLM tools. Writes are harness side effects. Full stop.

• Reads (LLM-callable): goto, get_stories, click, read_page, count_upvotes.
• Writes (harness-only): record_upvote, mark_verified, fill_credentials.

The rule is enforced at three layers:

1. Schema: the write actions aren't in the LLM's tool schema at all.
2. Pre-dispatch guardrail: even if a malformed tool call slips through, the guardrail refuses actions not on the read allowlist.
3. Dispatcher: even if it slipped through the guardrail, the dispatcher only routes read actions.

Defence-in-depth. A compromised LLM (prompt injection, model drift) has to break through three unrelated layers to write. Any of the three going wrong alone is a bug, not a breach.

Rule 8 — One reasoning pass, maximum output

The last principle came from realizing I was making the LLM reason twice about the same input.

The bug: I had a Python-side title-extractor (regex fast-path + LLM fallback) that ran before the agent loop. It reasoned about the user's message to find the target. Then the agent loop ran and… also reasoned about the same user's message to decide which tools to call.

Two reasoning passes. Twice the cost. Duplicated logic. Two places to be wrong.

The fix (this is Rule 6 applied): put the extraction inside the agent loop as a set_target tool call. Now the LLM reasons about the user's message once, produces a declaration, and continues in the same context. Every constraint keys off that single declaration.

Before / after cost math for a natural-phrasing run:

Scenario	Before	After
History-only question	0 extra + N loop iters	N loop iters
Quoted title	0 extra + N loop iters	N loop iters (+1 for set_target)
Natural phrasing	1 extra + N loop iters	N loop iters (+1 for set_target)

Same total calls, but the architecture halved in size. One less module, one less LLM API method, one less regex to maintain, one less validation to write. Simpler is cheaper — even before you count the dollars.

The anti-patterns I now sniff for

Every one of these bit me at least once. If you catch yourself doing any of them, stop:

• Prompt cooperation for correctness. Adding "please retry on failure" to the system prompt instead of building retry logic in Python.
• Circular trust. Letting the LLM define its own constraint. Validate against something outside the LLM's world.
• Mocking the layer under test. Testing the verifier by mocking the browser tool proves nothing. Fake at the protocol boundary.
• Bundling refactor with bugfix. "While I'm here, I'll also…". Two commits, always.
• Speculative abstraction. Adding a Protocol or config knob "just in case". Add it when the concrete need appears, not before.
• Silent failure. A DB write that returns success on error is a lie. Log at least; usually re-raise.
• Task-specific code in the orchestrator. HN-specific logic in a Harness class is a leak. Move it to a task module the harness delegates to.
• Magic constants scattered across modules. Timeout, retry cap, cost cap → one config file. Selectors, prompts, failure signals → live with the task.

What the final architecture actually looks like

For anyone building along, here's the layered picture:

The direction of arrows matters. Presentation → orchestration → guardrails → adapters. Never up, never sideways. Task-specific modules like the recorder are services the orchestration layer delegates to — the orchestration layer itself stays task-agnostic. Swapping HN for Reddit means writing a new recorder and prompt; the harness doesn't change.

The one takeaway

If you remember one thing from all of this:

Correctness lives in code. Policy lives in prompts. The LLM makes decisions, the harness makes guarantees.

Every rule above is a specific case of that. When you're deciding where a new behavior should live, ask: is this a guarantee I need to hold even when the model behaves badly? If yes, it's code. If no — style, tone, on-task refusal, response language — it's prompt.

Build the guarantees first. The LLM part is easy once you know the guarantees are there to catch it when it slips.

Bonus: the AI-installable version

While writing this up I also packaged these rules as a Claude Code / agent skill called harness-engineer. If you use an AI coding assistant that supports skills, you can install it into your own project and it'll auto-apply the same rules whenever you're designing or reviewing a harness — no re-reading the blog, no copying checklists around.

• What it is: a portable, imperative version of the 8 rules with red-flag triggers ("stop if you catch yourself doing X").
• What it isn't: tied to Python, Playwright, HN, or the specific codebase in this post. Same principles, generic scaffolding.
• Where to get it: SKILL.md source →
• How to install: drop the SKILL.md file into your project's .claude/skills/harness-engineer/ folder. Your assistant will pick it up on next session and auto-apply when the task involves "build an agent", "LLM tool calling", "guardrails", or similar phrases.

Think of it this way: the blog is why the rules exist. The skill is a coworker that reminds you to apply them. Use whichever fits your workflow.