When to let an agent act, and when to make it wait

Somewhere in the first week of adding an AI agent to a process, someone asks the obvious question: can it just do the thing, or does a person need to look first? That question doesn’t have a model-shaped answer. It isn’t about how good the model is this quarter. It’s about confidence, reversibility, and what a wrong action actually costs — worked out per step, not for the system as a whole.

Confidence is not the same as safety

A model can tell you how sure it is about an output. That number is useful, but it answers a narrower question than people assume: it says how often the model would guess the same thing again, not whether acting on that guess is safe. A model can be equally sure of a customer’s shipping address and of an order quantity, and still carry completely different risk on the two fields, because the two mistakes cost completely different things to fix.

So the first step is not to pick a confidence threshold and automate everything above it. It’s to separate “how sure is the model” from “what do we lose if it’s wrong” and treat them as two different inputs to the same decision.

What a wrong action costs, not how often it happens

Say a wholesaler’s inbox gets order emails, PDFs and the odd photograph of a handwritten pad. An agent that mis-reads a quantity and drafts the wrong order is a two-second fix for whoever reviews it. An agent that mis-reads a quantity and writes straight into the warehouse system has already told someone to pick the wrong pallet. Same error rate, same model, same input — different outcome, because one version of the system left a person between the guess and the consequence and the other didn’t.

This is the actual design question behind “human in the loop”: not whether a person is involved somewhere, but whether a person is involved before the action that can’t be quietly reversed.

Reversibility decides more than accuracy does

Some actions are cheap to take back: flagging an email, drafting a reply that nobody has sent yet, adding a line to a queue. Others are not: paying an invoice, deleting a record, telling a warehouse to ship. An agent that only ever produces the first kind of output can run with very little supervision, because the worst case is a person deleting a bad draft. An agent whose output is the second kind needs a person in front of every action, no matter how good its accuracy looks in testing.

In practice this means auditing the process for its irreversible steps first, before worrying about the model at all. Each one gets a person in front of it by default. The agent’s job upstream of that point is to make the person’s decision fast, not to make the decision itself.

Build the review queue for speed, not just correctness

Once you know where a person has to look, the queue they look at matters as much as the model behind it. A reviewer who has to re-read the whole source document every time will eventually stop reading it, and the review becomes theatre. A queue that is actually used shows the source and the draft side by side, puts the model’s own uncertain fields first, and lets a correction happen with a keyboard, not a form. We set this out at the level of an actual pipeline in the order intake agent write-up: a document comes in, the agent produces a draft order against the real catalogue, and the reviewer’s job is reduced to confirming or fixing a handful of highlighted fields, not retyping the order from scratch.

Every correction a person makes is also data. Captured and labelled, it tells you exactly where the agent is weak, which is more useful than any accuracy figure measured in isolation.

Prove it before you trust it

None of the above matters if you don’t know whether the agent is getting better or worse as you change the prompt, swap a model, or add a new document type. That needs a held-back set of real inputs with known-correct answers, run on every change, so a number — not a feeling from the last demo — tells you whether accuracy moved. Without that, “it seems to be working” is the only signal you have, and it’s the signal that fails quietly, well after the change that caused it.

The sequence we work to is: decide which actions are reversible, put a person in front of the ones that aren’t, build the queue so the person’s time is spent on judgement rather than re-reading, and measure the agent against a fixed test set before you ever widen what it’s allowed to do unsupervised.

A checklist before you start

• Which step in this process, if the agent gets it wrong, can be undone by the next person to look at it — and which can’t?
• Have you separated the model’s confidence score from an actual estimate of what a wrong answer costs?
• If a person has to review every output today, is their queue designed so reviewing is faster than doing the task from scratch?
• Do you have a fixed set of past, real inputs with known-correct answers that you re-run every time the prompt or model changes?
• What happens to a correction a reviewer makes — does it feed back into anything, or does the same mistake get made again next week?
• Is there any action in this process you’re planning to automate because it would be convenient, rather than because it’s actually safe to get wrong?