The Missing Anchor in AI Governance: Who Can Actually Stop AI

I keep coming back to one question. If the AI makes a recommendation and the person reviewing it thinks it’s wrong, what can they actually do? Having a human in the loop sounds reassuring. But I’m not sure it tells us very much on its own. The problem may not always show up as delay.

Sometimes things might actually get faster because people stop challenging the system. Nobody has to tell people to trust the AI. They just have to learn that questioning it is harder than going along with it. Somebody is still responsible for the decision, but how much power did they really have to change it? So who can actually stop the AI?

I’m Bess Obarotimi. I spend a lot of my time looking at decision environments — what happens between the governance an organisation puts in place and the decision somebody eventually has to make.

And there’s a particular question I’ve been thinking about as organisations bring more AI into those decisions. We hear a lot about keeping humans in the loop. And of course that matters. It’s reassuring, isn’t it? The AI hasn’t simply been left to make the decision by itself. There’s still a person there. Somebody can review what it produces. Somebody remains accountable. So it can feel as though we’ve dealt with one of the biggest risks. But I’m not sure we have.

Because I keep coming back to one question. If the AI makes a recommendation and the person reviewing it thinks it’s wrong, what can they actually do? I don’t mean what the policy says they’re allowed to do. I mean what happens in the real organisation, on an ordinary working day, when there are targets to meet, other cases waiting, somebody expecting an answer and perhaps a manager you’ll need to explain yourself to if you decide not to follow the normal route. What happens then?

Take a credit decision. An AI-supported system assesses an application and recommends decline. A member of staff reviews the case before the decision is finalised, and something doesn’t quite look right.

Maybe there’s more recent information that changes the customer’s circumstances. Maybe vulnerability is relevant. Maybe one of the data points looks inconsistent. Or perhaps there isn’t one obvious error at all. The reviewer simply looks at everything in front of them and thinks: I’m not sure this recommendation fits the wider picture.

This should be the point where having a person involved really matters. So what can they do? Can they ask for more evidence? More importantly, can they require it? Can they pause the process themselves? Can they send the decision back for reassessment? Can they override the recommendation? And if they can’t, who can? How quickly can that person be reached?

Having a human in the loop sounds reassuring. But I’m not sure it tells us very much on its own. What matters is whether that human can actually influence what happens next. And that becomes particularly interesting when you look at where financial services already is with AI.

In the Bank of England and FCA’s latest published sector survey, 84% of responding firms said they had an individual accountable for their AI approach. Seventy-five per cent were already using AI. More than half of the reported use cases involved some automated decision-making, while only 2% were described as fully autonomous. So this isn’t really a story about humans disappearing from organisations. They haven’t.

And it isn’t a story about organisations completely ignoring accountability either. They’re putting policies in place. They have model inventories, risk classifications, approval processes, oversight committees and named people who carry responsibility. Yet the Treasury Committee has still reported uncertainty about accountability under the Senior Managers and Certification Regime when AI is involved.

And I think that’s worth paying attention to. Because if somebody has already been made accountable, and there’s still a person in the decision process, why can accountability become difficult precisely when something needs to be challenged? One possibility is that we’re answering one question quite well — who is responsible? — while being less clear about another. Who can actually change the decision?

Go back to the person reviewing that credit application. Let’s say the organisation has formally given them permission to question the AI recommendation. Fine.

Now look at what the two available choices actually involve. Accepting the recommendation might take a few seconds. Challenging it might mean gathering more evidence, recording why you disagree, finding somebody who can reopen the case, waiting for another review and perhaps putting your own judgement more visibly on the line if you later turn out to be wrong. Both options technically exist. But they don’t feel the same to the person making the decision. And I think that matters.

Because if one route is fast, normal and easily defensible while the other is slower, more exposed and requires several extra steps, the organisation is already influencing which decision is easiest to make. This can happen without anybody deliberately designing it that way.

Nobody has to tell people to trust the AI. They just have to learn that questioning it is harder than going along with it. And people do learn. They learn which actions make their workload bigger. They learn which escalations annoy people. They learn what affects their targets. They see how managers respond when someone repeatedly raises exceptions. And they notice what happens when a colleague ignores the system recommendation, uses their own judgement and gets it wrong.

Over time, that changes behaviour. The person may still be sitting there reviewing every decision. The organisation may still be able to say that a human was involved. But the reality of that involvement can become thinner than it first appears. And this is why I don’t think this is simply a culture issue or a people issue. It belongs inside AI risk management. Because it affects whether the human control everybody is relying on can actually operate when it’s needed.

A governance framework can tell you who owns the model. It can tell you who is accountable for the activity. It can tell you who reviews an output.

But those things don’t automatically tell you who can stop a live process, who can insist on reassessment or who can say: no, there’s enough uncertainty here that we need to look at this again. This is the distinction I’ve been making between formal authority and what I call executable authority. Formal authority is what the organisation says you’re allowed to do. Executable authority is whether, when the moment comes, you can actually do it.

And even that doesn’t solve everything. Someone can have the right to intervene and still not have enough information to make a good judgement. They may not understand the limitations of the system well enough. They might not have enough time. They might lack independence.

They may simply be dealing with so much work that meaningful review becomes unrealistic. Automation bias doesn’t disappear because somebody has been given permission to challenge the machine. And performance measures can quietly work against good judgement too. If every challenge makes a case take longer, that matters. If overrides receive considerably more scrutiny than accepting the recommendation, that matters too.

Current regulatory expectations already recognise parts of this. The ICO has said that human involvement in AI-supported decisions should be active rather than tokenistic. Reviewers need to be able to meaningfully influence the outcome and have the authority and competence to go against an automated recommendation.

Its guidance also looks at things such as knowledge, experience, independence, workload, training and what happens when decisions are overridden. The EU AI Act similarly links effective human oversight of high-risk systems with the competence, training and authority required to intervene. So simply having a person somewhere in the process isn’t the end of the story. The environment around that person matters too.

This is actually one of the things I’m researching through my work on decision environments. I want to understand what happens inside real organisations, not just what policies and frameworks say should happen. So if you work somewhere AI is influencing consequential decisions, I’d really value your experience. What happens when something doesn’t look right? Can you challenge it? And what happens when you do?

There’s a short questionnaire linked with this commentary. If you have relevant experience, I’d be very grateful if you took part.

Now, take the same problem and move it further up the organisation. It gets more complicated. A model might be developed by one team. Part of it may come from a third-party supplier. Another team validates it. Risk has a role. Compliance has a role. Operations actually uses it. Data specialists may understand parts of the system that the people dealing with customers never see. And somewhere in that structure there may still be one senior manager who is ultimately accountable for the activity.

That division of work isn’t necessarily wrong. Large organisations have to divide work. The difficulty is what can happen between all those different parts.

Knowledge can sit in one place. The ability to make a particular decision can sit somewhere else. And the consequences can eventually land with somebody else entirely. Somebody is still responsible for the decision, but how much power did they really have to change it?

That question matters at senior level just as much as it does for the person reviewing an individual customer case. If the accountable person cannot see the relevant evidence, cannot tell where challenge is breaking down or cannot quickly get the right people to intervene, then accountability and practical control may not be sitting in the same place.

That becomes especially interesting in financial services because the FCA has deliberately chosen not to create a separate AI rulebook. Firms are expected to work with existing frameworks, including the Consumer Duty, SM&CR and established governance and controls.

So organisations cannot simply wait for a new AI rule to tell them exactly who should be allowed to do what in every possible situation. They have to translate existing accountability into actual decisions. And that means being much more precise about authority. Not just: who has authority? Authority to do what? Can you pause? Can you require more evidence? Can you send something back? Can you override? Can you escalate? Can you stop the use of a system if you see the same problem happening repeatedly?

And under what conditions can you do those things? A committee can have oversight of a system and still be useless for resolving a live case at 2.30 on a Tuesday afternoon. A frontline member of staff can spot the problem and still have no way of changing it. A senior manager can carry accountability and still be dependent on four different teams to understand what’s happening.

But there’s another part of this that I think we’re not looking at enough. A lot of AI governance understandably follows the lifecycle of the system. How was it developed? How was it validated? How was it deployed? How is it monitored? When should it be retired? All of that matters.

But a decision has a lifecycle as well. Evidence comes in. The system classifies something, predicts something, ranks something or interprets what it sees. It produces an output or recommendation. Somebody then accepts that recommendation, questions it or brings something else into the decision. An exception may be raised. Risk gets accepted somewhere. And eventually the result reaches a customer, employee, patient, citizen or counterparty.

A model can work exactly as designed while the decision around it still fails. And that’s quite an important distinction. The system itself may not have malfunctioned at all. Maybe it processed the information it was given perfectly well.

But perhaps the information was incomplete. Perhaps someone noticed contradictory evidence and couldn’t get it considered. Perhaps uncertainty was recognised but nothing happened. Perhaps the escalation process simply took too long. Or perhaps the person who saw the problem didn’t really have enough authority to alter what happened next. That isn’t necessarily a model failure. It’s a failure somewhere around the decision.

And if you’re a CRO or COO, you may not initially see any of this as an AI issue. You may see cases moving sideways between teams. You may see people constantly asking managers for reassurance. Risk and compliance might keep getting pulled into the same operational questions.

Exceptions might sit waiting. Processing times might increase. That looks like friction. And those kinds of symptoms are relatively easy to notice. But the opposite possibility is the one I find more interesting. The problem may not always show up as delay. Sometimes things might actually get faster because people stop challenging the system. Imagine that for a moment. Throughput improves. Escalations fall. Cases close faster. The numbers look better. And maybe they really are better. Perhaps the AI system has improved the quality of the decision and there genuinely isn’t as much need for intervention. But there’s another possibility.

Maybe people have simply learned that challenging the recommendation takes too long. Maybe they know it creates more work. Maybe they no longer believe anything will change if they raise the issue. So the number of challenges falls. Those two situations can produce the same operational metric. Fewer escalations. But one means the system is working better. The other means people have stopped questioning it.

That’s why I’d be careful about treating efficiency on its own as proof that the decision environment is healthy. Sometimes the interesting question isn’t why things are slow. It’s why everything suddenly became so easy.

A quick favour before I carry on. If you’re finding these commentaries useful, please subscribe. It helps the work reach more people who might be able to use it, contribute to it or challenge it. And I’d really appreciate it.

None of this means existing AI governance has failed. I think that’s important to say. Model assurance still matters. Data controls still matter. Risk classification and monitoring matter. Senior accountability matters. The organisation may have very good technical governance. The model itself may be performing exactly as expected.

What changes with AI is the speed, scale and sometimes the apparent certainty surrounding a recommendation. Something that used to give people time to talk, check something informally or ask another person can move from evidence to recommendation to action very quickly. And that changes what the surrounding organisation needs to be able to do.

So if I were trying to understand this inside an organisation, I wouldn’t begin by reviewing every AI system in the business. I’d start with one consequential decision. Just one. Follow it from beginning to end. Where does the evidence come from? Where does AI enter? What does it change? At what point does its output begin influencing what a person does next? And where does responsibility eventually land? Then make the scenario slightly uncomfortable. Assume the recommendation looks wrong. What happens now? Who can bring in contradictory evidence? Who can require reassessment rather than politely suggest it? Who can pause the process? Who can override?

If the same problem starts appearing across several cases, who can say: stop using this until we understand what’s happening? And then there’s the part I think is particularly revealing. How easy is any of that to do? Does somebody need a particular manager to be available? Does escalation damage a performance measure? Does the formal route take so long that nobody uses it? Do people follow the process that’s written down, or do they call somebody they happen to know because that’s the only reliable way to get something changed? Does challenging the system require an unusually confident employee?

Those answers tell you something the organisation chart won’t. They show you how authority actually behaves inside the decision environment. And this isn’t only a banking question. You can ask the same kinds of questions around insurance claims, customer acceptance, fraud investigations, healthcare triage, employment decisions, public services and plenty of other areas where systems increasingly classify, predict, rank or recommend. The regulations will differ. The consequences will differ. The level of human involvement that makes sense will differ too. But the basic question travels surprisingly well.

When AI changes the evidence people see, the speed at which the decision moves or the apparent certainty surrounding a recommendation, can the person expected to use judgement still affect the outcome? That, for me, is the part worth examining. Because the organisation may have a human in the loop. It may have somebody clearly named as accountable. It may have good policies. Its committees may be doing exactly what they’re supposed to do. And the model might be working perfectly well. But when somebody looks at the recommendation and says, “I don’t think this is right”, what happens next? Can they change it? Can they stop it? Can they get somebody who can?

So who can actually stop the AI? I think that may be one of the missing pieces in AI governance. Not more authority everywhere. Not giving everybody permission to override systems whenever they disagree. And not another layer of approval. Something more practical than that. Making sure that where human judgement is supposed to matter, the person exercising it has the information, support and usable authority needed to do something. Otherwise we can end up in a strange position. The human is still there. The responsibility is still there. But the ability to alter the outcome may have become much harder to see.

And that, I think, is the distinction worth paying attention to as AI becomes more deeply embedded in consequential decisions.