Where Human Judgment Still Wins

Published on LinkedIn Sep 10, 2026

Also published on LinkedIn.

"Put a human in the loop" has become the corporate comfort blanket for AI. It sounds responsible. It photographs well in a deck. And according to a growing body of research, it often makes the decision worse. That is the part most leaders skip.

I am COO of BSC Analytics. I run the operating company. My job is not to be impressed by a model. It is to keep delivery honest when tools meet real work — incomplete inputs, contested ownership, and people who have to live with the outcome on Tuesday. The useful question is not whether AI is "taking" judgment. It is where judgment still earns its keep — and how we stop pretending a rubber stamp counts as one.

The surprising part of the research

In late 2024, researchers at MIT’s Center for Collective Intelligence published a systematic review in Nature Human Behaviour covering 106 experiments and 370 effect sizes on human–AI combinations. Thomas Malone called their headline finding the most surprising result: on average, human–AI teams performed worse than the best of humans alone or AI alone. The combination lost to the stronger solo player.

The pattern underneath matters more than the average. Combinations tended to lose ground on decision tasks — exactly where companies invent a human "monitor." They gained ground on creation tasks, where people and generative systems iterate together. When humans already outperformed AI on a task, adding AI usually helped. When AI already outperformed humans, adding a human who could not tell when to trust the model usually hurt.

In one MIT example, AI alone hit about 73% accuracy detecting fake hotel reviews. Humans alone were around 55%. Humans plus AI landed at about 69% — better than people, worse than the model. People were not bad at reading reviews. They were bad at knowing when to override the algorithm. That is not an argument against humans. It is an argument against decorative oversight.

Where judgment still wins

Human judgment still wins where models are structurally weak: context that never made it into the training set — the exception that looks like noise until you know the customer, the contract, or the room; tradeoffs with no clean objective function (speed versus risk, tone versus precision, what we are willing to own if we are wrong); taste — the difference between a technically fine answer and the right one for this moment; and accountability — someone has to explain the decision after the fact, not "the model suggested," but "here is why we chose this."

AI raises the floor on volume, pattern, and draft quality. That is real. Erik Brynjolfsson, Danielle Li, and Lindsey Raymond studied more than 5,000 customer-support agents given a generative assistant trained on successful conversations. Issues resolved per hour rose about 14% on average, and roughly 34% for newer and lower-skilled agents. The tool spread what good agents already did. It raised the floor under the conversation. It did not replace it. That is complementarity: AI does what it does best so people can spend more judgment on what only people can do.

The loop that actually works

A real human-in-the-loop from an ops seat is not: model decides, human nods, everyone sleeps better. It looks more like this:

Independent judgment first when stakes are high. Automation-bias research keeps showing the same failure — people over-trust algorithmic recommendations, especially when the error is large and the answer arrives before the human has formed a view. If your loop starts with the model, you have already anchored the room.

Clear decision rights. Who can stop it? Who accepts done? Who gets paged when elegant output meets a messy Tuesday?

A kill criteria for the automation itself. Confidence thresholds, exception classes, customer impact. If you cannot name when the human must override, you do not have a loop. You have a hope.

Skill where the human is still better. The MIT pattern is blunt: if people cannot outperform the model on the underlying task, they are often also worse at knowing when to trust it. Train for discrimination, not ceremony.

Creation and iteration as the sweet spot. Draft, challenge, rewrite. That is where human–AI combinations have shown real synergy — not as a moral preference, but as an empirical one.

A line I will own

A human in the loop who only confirms the model is not judgment. They are a witness with a badge. Judgment is the moment someone can still say no — for a reason the organization will stand behind.

What this means

AI is not replacing the need for human judgment. It is raising the cost of fake judgment. The companies that get this right will not brag about how many humans sit near their models. They will redesign the work so models raise the floor on volume and pattern, and humans spend scarce attention on context, exceptions, taste, and ownership. The rest will keep adding "human in the loop" to the slide — and wonder why the combination keeps underperforming the better of the two parts.

The lingering thought is simple: the future of work is not humans versus AI. It is whether we are brave enough to put judgment where it still changes the outcome — and honest enough to remove it where it only makes us feel safer.