Essay
Who is accountable when the model is wrong?
AI can be Responsible for the work. It can never be Accountable for it. Almost every hard decision in an AI product falls out of taking that distinction seriously.
"Human in the loop" is the phrase everyone reaches for, and it is not worth much. It asserts that a person is present. It says nothing about whether that person can see what happened, understand why, or do anything about it. A human who rubber-stamps a screen full of confident-looking suggestions is in the loop. They are also, in every way that matters, not.
I find it more useful to borrow a lens from project management. RACI asks four questions about any piece of work. Who is Responsible for doing it? Who is Accountable, meaning who answers for it when it goes wrong? Who must be Consulted, and who must be Informed?
The reason it is a good fit is a rule the framework has always had: responsibility can be shared, but accountability sits with exactly one person, and it cannot be delegated. You can hand someone the work. You cannot hand them the blame.
Which produces a clarifying result. A model can be Responsible. A model can never be Accountable, because accountability requires something you can hold to account, and there is nothing there. So on every AI feature, some named human is Accountable whether you designed for it or not. The only question is whether they know, and whether the system gives them any way to act on it.
The failure
Accountability leaks by default
Here is how it goes wrong, and it is never a decision anybody makes.
You ship a feature where the model proposes something and a human approves it. Responsible: model. Accountable: human. Clean. Then the model gets good. The approvals become routine. The human starts clicking through them, because ninety-five percent of the time it is right and they have other work. Nothing in the system changed. But the human is no longer exercising judgment, they are providing liability cover, and the model is now accountable in practice.
Nobody decided that. It happened because approving was cheap and scrutinizing was expensive, and people do the cheap thing. If your design relies on a human being conscientious at 4pm on a Friday, you have not designed anything.
In practice
What this forces you to build
I built a system that texts a construction project manager every evening and turns their plain-English reply into schedule changes. Real money, real deadlines, and a user who will not open an app.
Assigning the roles properly is where it gets useful, because the roles belong to parties, not to mechanisms.
Task: Update the schedule from the PM's daily check-in reply
| Role | Who | Why |
|---|---|---|
| Responsible | The extraction pipeline | It does the work: reads the reply and produces the change. |
| Accountable | The project manager | Their schedule, their client, their money. One person, undelegable. |
| Consulted | The PM, when the model asks a clarifying question | Two-way. The bot texts back rather than guessing, and the answer feeds a fresh extraction. |
| Informed | The PM, by receipt, after an auto-apply | One-way. Told after, with the ability to reverse it. |
Once the roles are assigned, the confidence threshold stops looking like a tuning parameter. It is the line between the system acting and the system withholding. Above it, the pipeline applies the change and sends a receipt. Below it, nothing is written and the proposal is parked for review.
Most teams arrive at that number by feel and then treat it like a retry count. It is not a retry count. It decides which class of decisions a machine may make on a person's behalf without asking, and moving it two points moves that line for every message that arrives afterward. It looks like an engineering knob. It is closer to an org chart.
Notice that Consultedis not the threshold. Consultation is a separate mechanism: when the model cannot tell what a message refers to, it does not guess and it does not quietly drop it, it texts back a question and folds the answer into a fresh extraction. That distinction matters, because a low confidence score and an honest "I do not know" are different signals, and only one of them should start a conversation.
Two things follow from the rest of the table. The Responsible party has to earn the right to act unilaterally, which is what the self-consistency vote is for: three concurrent runs, and a proposal survives only if two of them independently agree. And because being merely Informed is only safe when reacting is cheap, the reversal has to be exact. Every applied change writes the prior state into an append-only log, inside the same database transaction as the change itself. Not eventually, not in a hook. In the transaction, so it cannot be half-done:
before_state = {
"status": milestone.get("status"),
"inProgressAt": milestone.get("inProgressAt"),
"completedAt": milestone.get("completedAt"),
}
# written to the append-only log in the SAME transaction as the update,
# so rollback is an exact inverse, not a guess.Informing them is a text message, not a dashboard. The channel they are already in, not a notification badge in an app they do not open. And the veto has to be cheaper than acquiescence, so they reply undo. One word. Or undo 1 3 to reverse specific lines of the receipt they just read.
_UNDO_RE = re.compile(r"^undo(?:\s+(all|[\d\s,\-]+))?$")That last detail is not an implementation note. It is the argument. A person trying to reverse a mistake made by an AI should not have to get past another AI to do it. The undo path is deterministic precisely because it is the path that has to work on the day the model is behaving badly.
The hard part
Confidence is not correctness, and thresholds cannot save you
The system marked a live, in-progress permit review as not-applicable. It was wrong, it mattered, and it was not a low-confidence guess that squeaked past the gate. The model was confident. All three runs agreed with each other.
This is the thing people miss about self-consistency voting, and I missed it too. Agreement between runs is not independent evidence. If the failure is in how the model reads a particular situation, all three runs share it, and they will agree enthusiastically on the wrong answer. Consensus measures stability, not truth.
So the fix was not a higher threshold. You cannot tune your way out of a failure mode that lives inside the thing you are tuning. The fix was a deterministic rule, in code, above the model, that refuses to mark an in-progress milestone not-applicable no matter what the model says or how sure it claims to be.
This is where a security background turns out to be unreasonably useful. It is the same instinct as never trusting a client-supplied value: it does not matter how well-behaved the input usually is, you constrain it at the boundary, because the case you are defending against is the case where your assumptions are wrong. A model is an untrusted input that happens to be very persuasive.
The payoff
The table tells you what to build next
Here is the part I did not expect. Go back to that RACI table and look at who is in it. The project manager is Accountable, Consulted, and Informed. Three of the four rows are the same person. The only party who is not them is the software.
A degenerate RACI is a product finding, not a formatting problem. It says the system is an excellent note-taker for one person. Clara is still asking the PM what happened, so the PM still has to know, so the expensive part has not been removed. It has only been made cheaper to write down.
The roles only separate when the subcontractor reports for himself. And that is a different product, not the same one with a longer contact list, because the moment the reporter is not the accountable party, their incentives come apart from the truth. A sub has reason to call the drywall done on Thursday. The PM is the one who answers for it being done.
Which gives you the design rule before you write a line of it: a subcontractor's message is evidence, not state. It can raise confidence, corroborate a photo, or prompt the PM. It must not silently move a milestone. That constraint came out of the framework, not out of an incident, which is the entire reason to bother with the framework.
Where I land
None of this is anti-AI. The system I am describing applies most of what it extracts automatically, and it captures schedule state that would otherwise not have been written down at all, because nobody had time to write it. The model does real work.
But it does that work as a Responsible party, under a human who is Accountable, with a log that shows what it did and an undo that costs one word. The accountability did not survive by accident. It survived because it was designed in, and because the reversal path was built to work on the model's worst day rather than its average one.
The whole system is written up as a case study.