The Decision Nobody Made

We can end up going in a particular direction without being able to pinpoint who actually made the decision in the live process.

Sometimes the organisation knows who owns every part of the decision and still cannot tell you who made the decision.

The person who carries the accountability may not be the person who carried the authority.

You can leave a human with the right to intervene while slowly removing everything they need to intervene well.

More human authority is not automatically better governance.

Where exactly did the decision happen?

I’m Bess Obarotimi.

I spend a lot of my time looking at decision authority. I’m helping organisations ask questions such as who can act, who can challenge, who can change direction, and what has to be true around a decision for that authority to be meaningful — to actually work.

This week I thought I’d take one step before those questions. Well, actually, I’ve spent the past two weeks looking at this. I decided to do the research before I put out the commentary.

So before we ask who has authority over a decision, we can ask:

Where exactly did the decision happen?

Take software engineering as one example.

An AI coding agent produces a change. Another agent reviews it. Automated tests check whether it works. Security tooling may assess it. Everything looks good. It’s passed.

A human sits somewhere above that process, but there’s a lot going on, far too much activity for that person to inspect every individual output.

So the work moves on.

And that same basic pattern could exist in credit, insurance, healthcare, compliance or any environment where several systems and people contribute to one consequential outcome.

But what about when something goes wrong?

Can we determine who made the decision that the work was good enough?

The first agent?

The review agent?

The person who designed the testing threshold?

The person who gave the agents permission to operate?

The human supervising the overall process?

Or did no single person actually make that decision at all?

I heard Jason Stanley, Head of Applied AI Research at ServiceNow, talking about something very close to this on Freshfields’ The AI Drop.

[AUDIO CLIP — JASON STANLEY, THE AI DROP, 12:34]

“Distinguish governance of AI from governance in an era of AI.”

That’s an important distinction to consider.

There’s a difference between governing the AI — the actor — and governing the work that’s now being produced through a combination of people, agents, workflows, permissions, models and controls.

Stanley’s point is that organisations already have ways of governing work. Who is allowed to review something? What happens before it can move forward? Which credentials or permissions are required?

But agents change how that work gets done. They can operate at a speed, scale and volume humans simply cannot.

And if you start looking at the work and not just AI, it may not be as straightforward to pinpoint who made the decision.

Something produces the work.

Something else assesses it.

Another system tests it.

Somebody established the permissions.

Somebody established the threshold.

Along the way, the work was acceptable enough to proceed.

We can end up going in a particular direction without being able to pinpoint who actually made the decision in the live process.

Another version of the problem that has come up in conversations I’ve had with people who work in risk, governance and data is what happens when one consequential decision crosses several functions and those functions don’t agree.

One practitioner I spoke to described something that, from what I’ve witnessed, can sound completely normal to anyone who works in a large organisation.

So the board approves the framework.

Management owns the process.

Risk or Compliance has challenge rights.

A business executive is accountable.

On paper, great.

But when the recommendation becomes contested, you realise that the word “ownership” carries a lot of weight.

Who recommends, who approves, who vetoes, who makes the exception, who can stop the decision altogether — it gets a bit more complicated.

Those don’t necessarily carry the same authority.

Another risk practitioner put it slightly differently.

And he said something that I thought was really important.

When those functions disagree, sometimes the decision is effectively made by whichever function has the strongest governance mandate.

Not necessarily the person named as accountable.

Not the person supposedly making the final call.

The function with enough organisational weight to make its judgement stick.

Now, he was talking about what he observes in risk management. He was very clear that he hadn’t seen this specifically in AI-supported decisions.

But the underlying question still applies.

If one function effectively determines the outcome, does accountability move with the authority?

Because it may not.

And if it doesn’t, I think we’ve got something worth looking at much more carefully.

Sometimes the organisation knows who owns every part of the decision and still cannot tell you who made the decision.

Governance can look great on paper and still become much less clear in a live decision.

If you follow me on LinkedIn, you’ll know I’ve had my head down taking notes in research papers. I read Åke Elden’s paper, When Responsibility Fails to Arise.

He makes a stronger argument than saying accountability becomes distributed or unclear.

He questions whether some algorithmic systems fail to produce the sort of identifiable action that responsibility normally attaches to in the first place.

For him, three things matter.

Can we identify a sufficiently bounded decision or act?

Can that act meaningfully be attributed to somebody?

And is there actually a forum in which that decision can be questioned, defended or contested?

He calls those discreteness, authorship and publicness.

Now, this is a conceptual argument. It isn’t an empirical rule that we should assume applies to every automated system.

And I’ll be honest, I squinted a little at the idea of there needing to be a forum.

One of the reasons organisations adopt AI is speed. We obviously can’t create a committee meeting every time an AI system produces an outcome.

Decisions have windows.

So perhaps the question isn’t whether every automated decision needs a forum. It’s whether, for the decisions that matter, there is somewhere for challenge to actually change the outcome before that window closes.

Something else I was reminded of in the paper is that an output is not necessarily a decision.

In a real workflow:

A model produces a score.

A threshold turns that score into a category.

The category triggers a workflow.

The workflow initiates an action.

A human sees the result.

Maybe an appeal exists later.

Score.

Threshold.

Category.

Workflow.

Action.

Appeal.

Which one was the decision?

At what point did somebody actually exercise judgement and turn all of that into an authoritative determination?

Because if we can’t identify that point, asking who is accountable for the decision might already be starting too late.

There’s a reason I’m spending so much time on this.

This is one of the questions I’m currently testing through my work on decision environments.

I’ve been speaking to people working across risk, governance, data and AI, and I’m particularly interested in people dealing with consequential AI-supported decisions.

I’m trying to understand some of the things we’ve already spoken about.

Where does an AI output become a decision?

Who can materially alter it before it takes effect?

When several functions disagree, whose judgement actually determines what happens?

Can the person named on paper still be made accountable for the outcome?

There’s a short anonymous questionnaire linked in the footnotes of this commentary.

I’d love to know all the awkward examples.

If your governance works extremely well, tell me that.

If you’ve found that giving humans more authority actually made the decision worse, let me know that too.

If the problem I’m describing is already dealt with perfectly well by existing governance in your organisation, that is useful evidence.

As AI becomes increasingly embedded in our workflows, there’s going to be so much to discover. And hopefully I’ll get to share some of that with this audience in a way that helps all of us in our work.

OK, so let’s talk about something that some of you may find even more uncomfortable to look at.

So far we’ve been struggling to locate the person who made the decision.

And yet organisations can become remarkably good at finding a person once something goes wrong.

Madeleine Clare Elish coined the phrase moral crumple zone.

The idea is that control across an automated system can become distributed between software, designers, managers, operators, organisational processes and other actors.

But when something fails, responsibility can collapse back onto the human closest to it.

Here, the system gets more complicated.

Control spreads out.

And somehow the person at the edge of the system becomes incredibly visible when blame needs somewhere to land.

Elish isn’t saying this happens in every automated failure. Her own work gives counterexamples.

But it’s a possibility, and I think we need to look at it carefully.

Because the person who carries the accountability may not be the person who carried the authority.

And there’s another part of this that I think is even more important for AI governance.

We spend quite a lot of time asking whether the human has permission to intervene.

Can they override?

Can they stop the system?

Can they depart from the recommendation?

Fine.

Suppose they can.

What if they no longer know enough to do it well?

What if automation handles so much of the ordinary work that the human has less situational awareness?

What if a skill they once exercised every day is now used once every few months?

What if the system is acting across hundreds or thousands of cases and the human is technically supervising it but practically nowhere near most individual determinations?

That problem came up later in the same Freshfields conversation.

Anna Gressel was talking about what happens when the familiar instruction to keep humans in the loop stops being something humans can realistically do for every output.

[AUDIO CLIP — ANNA GRESSEL, THE AI DROP, APPROX. 20:15–20:43 — FROM “WHAT’S THE RIGHT CHECKPOINT ARCHITECTURE?” THROUGH “HUMANS THAT SIT OVER THOSE SUPERVISORY MODELS?”]

And that’s interesting, because the human hasn’t disappeared.

They’ve just moved.

They may now sit above a system that is itself reviewing another system.

So saying “the human is in the loop” tells us almost nothing about where the judgement is happening.

The human may still have formal authority.

But they’re further away from the individual decision.

And that changes the question again.

You can leave a human with the right to intervene while slowly removing everything they need to intervene well.

Permission isn’t enough.

You need information.

You need capability.

You need enough understanding of what the system is doing.

And you need time.

That last one keeps coming back in the practitioner conversations I’m having.

Everything has to happen while there is still time.

I actually wrote about this recently on LinkedIn in a post called “What happens when an experienced human thinks the AI is wrong?” I’ll put the link in the show notes if you want to have a read.

Because the question isn’t only whether they can challenge it. It’s whether that challenge can reach somebody who can act before the decision window closes.

I was discussing escalation measures recently with an AI transformation leader.

Organisations already track things like escalation SLAs, manual review turnaround, exception ageing, resolution time and sometimes override rates.

All useful.

But his point was that those measures tell us whether something eventually got resolved.

They don’t necessarily tell us whether somebody with executable authority could alter the outcome before altering it stopped mattering.

And now you may say the answer is obvious.

Give the human more power.

Let them override.

Make sure they can stop the AI.

Except there’s a problem with that answer too.

There was a field experiment published this year that looked at 553 workers making inventory decisions alongside an algorithm in a smart-vending operation.

Workers who were given unrestricted freedom to override the algorithm actually reduced sales.

So they tested a more constrained form of human override which performed better.

The humans still contributed information the system didn’t have.

Their judgement still had value.

But unlimited discretion wasn’t the best arrangement.

And I think that matters enormously.

More human authority is not automatically better governance.

The design problem is much more precise.

What authority should exist, for whom, over what, and at what point in the decision?

Maybe a human should be able to reverse some outcomes immediately.

Maybe another type of departure should require evidence.

Maybe some decisions should be highly automated.

Maybe another class of decision needs a deliberate pause before the output becomes effective.

Maybe the AI should be free to act right up until a particular consequence, risk level or exception appears.

Different authority for different moments.

That takes a lot more planning than simply designing the best system and assuming we’ll be okay because there’s human oversight.

And there’s something else that’s worth thinking about.

What if some of the most important decisions are being made much earlier than the final workflow suggests?

Somebody decides what the system is being optimised for.

Somebody defines what counts as good performance.

Somebody sets the threshold.

Somebody decides that productivity matters alongside compliance — or perhaps, without meaning to, gives productivity far more weight.

Somebody decides what information the model will see and what information it won’t.

By the time a human sees the recommendation, somebody has already made a series of decisions about what “good” is allowed to mean.

By the time the frontline human encounters the recommendation, a great deal of the decision environment has already been shaped.

So perhaps we also need to distinguish the person who makes a decision from the people who shape what can be decided.

That’s something I’m not ready to turn into a grand conclusion.

But I do think it changes how I want to map decisions.

I increasingly want to trace four things separately.

First:

Where was the outcome actually determined?

Not where the governance document says the decision sits.

Where did the thing that materially changed what happened occur?

Second:

Where did effective authority sit?

Who could actually alter, stop, permit or redirect the outcome?

Third:

Who was expected to answer for it?

That’s accountability.

And fourth:

Where did the consequence eventually land?

Who loses the customer?

Who carries the regulatory exposure?

Who has to explain the decision afterwards?

Who gets blamed?

Those four places might be the same.

That would certainly make life easier.

But some of the more interesting decision environments may be the ones where they aren’t.

The system helps determine the outcome.

Risk effectively controls whether it can proceed.

A business executive carries the accountability.

And the frontline person who interacted with the case becomes the person whose judgement is questioned afterwards.

Or some completely different arrangement.

Now, I don’t want to imply that separation is automatically bad.

It may be completely appropriate. And in many cases it is.

Risk should have authority the business doesn’t have.

Compliance sometimes needs the ability to stop things other people very much want to do.

Technical teams need to control permissions that operational staff shouldn’t be able to change.

Distribution isn’t necessarily the problem.

The more interesting question is:

What happens to that distribution when the decision becomes contested?

Because that’s when all the labels get tested.

And there’s a fairly simple way to look at this in your own organisation.

Take one real AI-supported decision.

One.

What does the system actually produce?

At what point does that output become consequential?

Who can change it before that point?

Who can stop it?

Who can make an exception?

Who controls the information, thresholds or options that shape what everybody else is able to decide?

When two legitimate authorities disagree, whose judgement actually determines what happens?

And if the outcome goes badly tomorrow, who will everybody expect to explain it?

Then compare the answers.

If they all point to the same place, good.

If they don’t, don’t immediately decide the governance is broken.

Because there’s one more challenge to my own argument that I think matters.

A data and AI leader I spoke to recently made the point that sometimes all of this is much simpler.

The strategic direction isn’t clear.

The operational strategy isn’t clear.

Ownership gets shuffled around because the organisation itself hasn’t properly decided what it is trying to achieve.

And I think that’s a fair challenge.

Sometimes what looks like a decision-authority problem is just weak strategy.

The Decision Environment Method has to be able to tell the difference.

Otherwise I’ve simply built a method that diagnoses every organisational problem as the problem my method happens to examine.

That isn’t useful.

Where I think it becomes much more interesting is when the strategic intention is reasonably clear.

The governance exists.

Responsibilities have been allocated.

People know what the organisation is trying to achieve.

And then the real decision behaves differently as soon as somebody disagrees.

That’s the point I want to see.

Because disagreement is where hidden authority becomes visible.

And that is also where this connects directly to the work I do with organisations.

I’ve opened a small number of AI Decision Authority Reviews.

It’s a ten-day review of one consequential AI-supported decision or agentic process.

Not the whole AI estate.

One decision.

I trace where the determination actually happens, who can intervene, what happens when different authorities conflict, and whether the people carrying accountability can meaningfully influence the outcome while there is still time to do so.

And importantly, I’m not going in assuming the answer is “give the human more control”.

Sometimes the right answer may be less human intervention.

Sometimes it may be a different intervention.

Sometimes it may be moving authority closer to the decision.

And sometimes the issue may turn out to have very little to do with the AI at all.

If you have one decision that looks tidy on paper and rather less tidy once people actually have to use it, the details of the review are linked with this episode.

If you want to learn more about the work I’m doing on the Decision Environment, follow me, Bess Obarotimi, on LinkedIn. Or visit bessobarotimi.com.

Why Decision Frameworks Cannot Guarantee Responsibility