Hello, I’m Bess Obarotimi, and I work mainly around decision authority, AI data governance and AI responsibility.

And today I want to remind us how human oversight is not human authority, and discuss why that’s important.

So let’s do this sort of imaginary exercise.

I want you to imagine that you’re responsible for reviewing AI-supported fraud decisions in a financial institution.

This is a job you’ve done for years. You understand customer behaviour, the risks, and you’re pretty good at identifying sus activity.

Maybe you don’t need to imagine at all. Maybe that’s your job.

Then after lots of investment, promotion, company promotion and buy-in, the company successfully deploys an AI system to assess thousands of transactions.

The AI is great. Makes the job so much easier and performs incredibly well.

Your job is checking the recommendations, which you do.

And over the months, you get really confident in the system. You begin relying on it.

And in some ways, you’re supposed to be relying on the system. It’s the reason the organisation invested in the first place.

But think about it.

If AI gets a recommendation right 99 times, what happens the hundredth time the human thinks it’s wrong?

Will that person trust their experience, question the recommendation, or will they think, perhaps the system knows something they don’t?

And there’s so many other things that can happen at this point.

But the point is that having someone in charge of approving AI-supported decisions doesn’t necessarily mean their involvement makes the decision better.

Human in the Loop or Human at the Helm?

On the Harvard Business Review’s IdeaCast, Paula Goldman, Salesforce’s Chief Ethical and Humane Use Officer — that was a mouthful — talks about moving away from the idea of a human in the loop and towards the idea of having a human at the helm.

Paula Goldman — HBR IdeaCast (audio extract)

That phrase, though, human in the loop, got, I think, kind of misunderstood in this wave of AI. It kind of got framed like, okay, AI is going to draft and people are going to approve.

And so instead, we need a system where we are putting the right things for human judgement at the right time. That is what we mean when we say human at the helm.

Now, what she said there is right.

Because rather than just having someone there, a human involved in AI-supported decisions, we should examine exactly what they are contributing.

And honestly, it’s all psychological.

It’s called automation bias.

Automation Bias and Human Judgement

Automation bias occurs when a lot of trust is given to an automated recommendation, even when there’s information that suggests the person should be questioning it.

A Beginner’s Guide to AI explores this.

A Beginner’s Guide to AI — Automation Bias (audio extract)

The danger is not that AI systems always make mistakes. Humans make plenty of mistakes without technological assistance. The danger is that automation can change how carefully we think. When people believe a system is highly capable, they may reduce their own level of attention. They stop checking the details. They become less likely to question surprising results.

They may even lose some of the skills needed to recognise when the system is wrong.

Our trust in AI, I think, in my opinion, should be approached cautiously.

Not because it’s necessarily bad. Sometimes the system is better than the human.

You know, too much interaction could reintroduce the very errors and delays we’re trying to overcome.

But judgement is an area where humans could add value.

It’s just left to organisations to establish where and how they should add judgement, and whether they’re getting the adequate support and authority to do so.

Because there’s a huge difference between someone making an informed decision to agree with AI and someone approving the recommendation.

It might look the same in an audit log.

Approved.

But they are completely different decision processes.

Human Oversight in AI: Four Questions Organisations Should Ask

And this is what I would recommend organisations do to establish whether human oversight adds value.

Take one AI-supported decision in your organisation where a human is expected to review, approve or challenge, and ask four questions.

1. What is the human actually here to contribute that the system cannot?

Now, this answer should be more than simply approving the recommendation.

2. Does the human have the information they need to make that judgement?

Can they access the information easily, understand the system’s limitations, know when to challenge?

Because without the right information, there is inadequate human oversight.

3. What process occurs when the human disagrees?

Now, understanding this greatly influences the efficiency of the decision-making process.

And a lack of clear authority, a lack of understanding the process and how authority works in that process, could prove to be worse off prior to the use of AI.

4. Do we know whether the human’s involvement improves outcomes?

And this one is really important.

Does the human involvement make it better? Add value?

The question we’re asking: what is the human there to contribute that the system cannot?

And I must emphasise, not every decision requires more human scrutiny.

Maybe less is more. Maybe a more experienced person is required, or none at all.

What’s important is an understanding of decision authority within your organisation, so you can confidently say how decisions are made, where human judgement is necessary, and whether the decision environment makes that judgement usable.

The Decision Environment Method and AI Decision Authority Reviews

And I do this sort of work through the Decision Environment Method.

I also have a small number of AI Decision Authority Reviews open, where I examine one consequential AI-supported decision or process over ten days.

If you’re responsible for an AI-supported process and would like to make sure that oversight is doing exactly what it’s intended to do, the details of that review are linked with this commentary.

But I would like to leave you with this.

The next time somebody tells you that an AI-supported process has human oversight, ask them what that oversight is supposed to achieve.

Then ask how they know it’s achieving it.

Because having somebody approve an AI recommendation tells us that a human was involved.

It doesn’t necessarily tell us whether they contributed meaningful judgement.

Human oversight is not human authority.

And if we’re going to rely on human oversight as a safeguard, I think we need to be able to demonstrate what that safeguard actually does.

If you’d like to follow my ongoing work on decision authority, you can find me, Bess Obarotimi, on LinkedIn or visit bessobarotimi.com.

If you’ve liked what we’ve discussed today, do follow or subscribe to this podcast.

It helps the commentaries reach more people who are interested in AI governance and AI responsibility.

Thank you.

Show Notes

About This Episode

Does having a human in the loop mean an organisation has meaningful control over AI-supported decisions?

In this commentary, I explore human oversight in AI, automation bias and why human involvement doesn’t necessarily mean better decisions.

I also share four practical questions organisations can use to assess whether human oversight is adding value.

Podcasts Referenced

Harvard Business Review — HBR IdeaCast

How AI Is Changing Talent, Not Just Tasks, featuring Paula Goldman, Salesforce’s Chief Ethical and Humane Use Officer.

Listen to the episode

A Beginner’s Guide to AI

Automation Bias: Why Human in the Loop May Be a Dangerous Illusion

Listen to the episode

AI Decision Authority Review

I offer a ten-day review of one consequential AI-supported decision or agentic process, examining how authority, oversight and accountability work in practice.

Explore the AI Decision Authority Review

Connect

Connect with me on LinkedIn for more on AI governance, decision authority and AI responsibility.