The illusion of the human in the loop.

The human in the loop isn't a reviewer. It's an alibi.

Every AI deployment ends at the same reassurance. Keep a human in the loop. Someone reviews the output, someone approves it, a person stays accountable. The risk is handled. It’s the sentence that lets a board sign off, and most of the time it’s a story we tell ourselves.

Approval has a failure mode, and it has nothing to do with how careful or how senior the reviewer is. The first time you check the assistant’s work, you check it properly, and it’s good. The tenth time it’s good too, and the checking starts to feel like a formality. The formality becomes the habit. You stop reading to catch the mistake and start reading to confirm there isn’t one. It looks like the same act. It finds nothing.

None of this is a character flaw. It’s automation bias, a well-researched phenomenon: the more reliable a system runs, the less anyone scrutinises it, and a review that comes down to a click stops being a review. People defer to a recommendation more readily when they’re told an algorithm produced it than when the same recommendation comes from a person. The endoscopy study that found doctors got worse with AI found the reason in the same place: they trusted the flags until they stopped looking as hard themselves.

And it predates AI. In any organisation with enough approvers, sign-off decays into a rubber stamp, because everyone trusts that someone else really looked. Five signatures on a document is not five reviews. It’s often zero reviews with five names attached. The assistant just becomes one more name everyone defers to, except this one also wrote the document.

There’s a second decay, and it’s better hidden, because it doesn’t look like decay at all. It looks like engagement.

Watch a code review. The pull request collects twelve comments. Naming, formatting, a missing test, whether the helper belongs in this module. The thread is lively and the review feels thorough. Very few of the comments ask whether the change solves the problem, because that question is expensive. You’d have to hold the whole system in your head. A comment about taste, about how the code looks, costs nothing, so the discussion pools where the cost is lowest, and the volume of it gets mistaken for depth. A committee will approve a nuclear reactor in minutes and debate the bike shed for an hour, because everyone can see a bike shed. Reviews aren’t different. The visible layer gets the attention. The hard layer gets the benefit of the doubt.

AI makes this worse in a specific way. AI-written code arrives pre-polished. Consistent naming, clean style, tests included. The cheap comments, the ones the review actually ran on, are gone. What’s left is either the hard question the review was always avoiding, or approval. Most reviews choose approval, and the silence looks like the code getting better. AI didn’t break code review. It removed the decoration that was hiding what code review had become.

The deeper change AI brings is about who pays. Picture a king and his court. The advisors do the heavy lifting, draft the policy, run the numbers, carry in the scroll. The king reads the last line, nods, and the decree goes out under his name. That arrangement held for centuries for one unglamorous reason: the advisor who gave bad counsel lost his head too. The consequence was shared.

Now the advisor is an AI. When the advice is wrong, it feels nothing. It isn’t fired. It doesn’t lose the room. It doesn’t lie awake. The king does. And even if he wanted to hold his counsellor to account, there’s a newer model around the corner; the advisor that gave the bad counsel is retired within weeks either way, and its successor carries no memory of the failure. The whole point of keeping a human in the loop was that a human stays accountable, and that part is still true. The manager owns every plan he waved through. What changed is that he now owns plans he no longer fully understands, written by a counsellor who pays nothing when they fail. Accountability without comprehension isn’t oversight. It’s a name to attach to the failure.

I’ve approved plans from my teams without reading them, and sometimes I said so out loud: is this ready to be approved, because I’m not going to read it. They were ready to take the responsibility, so I was too. That wasn’t negligence. It was the court working as designed: the people who wrote the plan carried its failure with me, and they knew it when they answered. Ask the same question of an AI and the answer is always yes, and it costs the answerer nothing. The words are the same. The arrangement underneath them is gone.

The honest objection is that maybe it doesn’t matter. If the AI is right more often than the reviewer would be, deferring to it is the correct call, and the rubber stamp is just efficiency wearing a worried face. Sometimes that’s true, and one institution has it in writing. US Navy pilots launching off an aircraft carrier are ordered to keep their hands off the stick, because the flight computer sets the launch attitude better than any human can, and manual correction is what causes crashes. The flight manual says it plainly: stay out of the loop, monitor the sequence.

But look at how that decision was made. Tested, written down, bounded to the launch sequence, hands back on the stick the moment it ends. The Navy didn’t drift into trusting the machine. It decided to, and defined exactly where the trust ends and the pilot’s judgment resumes. That’s a designed handoff, and it’s honest, the same way telling your team you won’t read the plan is honest. The manager approving AI output has made no such decision. He drifted into the handoff while keeping the ceremony of review, and the ceremony hides the drift from everyone, including him.

And the drift matters, because the AI’s mistakes that count aren’t rare and random. They’re the confident, plausible, wrong ones, the exact kind a skimming human waves through. The review was the one thing meant to catch them. It’s still there in name, so everyone feels covered.

The fix isn’t more reviewers or a longer checklist. Both feed the reflex. It’s making the review effortful again, on purpose, and aiming it at the layer that matters.

Write the rejection before you read the recommendation. Before the AI hands you its answer, write down the one thing that would make you say no to it. Then read hunting for that thing, instead of reading for reasons to approve.

If you can’t name what would make you reject it, you aren’t the human in the loop. You’re the rubber stamp.

My bet: the first time your kill-test catches a flaw in an answer that looked ready to approve, you’ll never approve blind again.

“Human in the loop” was never a safeguard. It tells you who gets blamed, not whether anyone will catch the error. A real loop needs a person hunting for the reason to say no, who pays if they miss it. Short of that, the loop is open, and we’ve quietly agreed not to mention it.

There’s a bigger version of this question I’m not going to resolve here. Every review ritual we have, code review, sign-off chains, board pre-reads, was designed for human-produced work with human-shaped flaws: sloppy in visible ways, wrong in ways a colleague could spot from the surface. AI-produced work is polished in visible ways and wrong underneath. Whether any of our review rituals survive that in their current form is an open question, and I suspect most of them don’t.

So the question isn’t whether there’s a human in your loop. There’s always a human to blame. The question is whether that human still understands what he’s signing.

---

References

1. Complacency and Bias in Human Use of Automation: An Attentional Integration, Parasuraman & Manzey, 2010

2. Endoscopist deskilling after AI exposure, Budzyń et al., Lancet Gastroenterology & Hepatology, 2025

3. Expectations, Outcomes, and Challenges of Modern Code Review, Bacchelli & Bird, ICSE 2013

4. Parkinson’s Law or the Pursuit of Progress, C. Northcote Parkinson, 1957

5. The F/A-18 catapult procedure: US Navy F/A-18 NATOPS flight manual, via Lauren A. Kahn, “The Myth of the Human-in-the-Loop and the Reality of Cognitive Offloading,” 2025.