Why AI doesn’t need to be perfect to be trusted

As AI becomes part of courts, healthcare and other critical public services, the question isn’t whether it will make mistakes. It’s how we design systems that people can trust when the stakes are high.

In this episode of The Digital Lighthouse, Zoe Cunningham is joined by Alex Wolff, Public Sector Director at Softwire, to explore how governments can adopt AI responsibly without slowing innovation.

Drawing on more than 20 years delivering digital transformation across government and healthcare, Alex explains why comparing AI to perfection is the wrong benchmark, how humans and AI should work together, and why trustworthy systems are built around accountability, oversight and continuous improvement.

Hello, and welcome to The Digital Lighthouse, where we get inspiration from tech leaders to help us navigate the exciting and ever-evolving world of digital transformation. I'm Zoe Cunningham.

We believe that meaningful conversations can illuminate the path forward, helping us to harness the power of technology for innovation, scalability, and sustainability.

In this episode, I'm delighted to introduce Alex Wolff, who's the director for public sector health and transport at Softwire. Over the last 20 years, Alex has worked with government departments, the NHS, and private healthcare providers to design and implement intelligent digital solutions that make a positive difference for the people using them.

In this episode, we're going to talk about safety and security in critical public sector infrastructure projects. We'll cover how new and emerging tech is affecting the landscape and discuss the surprising conclusion that being safe doesn't always mean going slow. Alex, welcome to The Digital Lighthouse.

Thanks, Zoe. It's really nice to be here.

Over the last 20 years, you've worked across large-scale projects in the public and in the private sectors. So what's special about the public sector?

It's really hard to answer that without making it sound like the private sector isn't special in its own way as well.

Yeah, right.

Because I've really enjoyed both. Actually, doing the two together has been really interesting because I think there's a lot they can learn from each other. But I think for me, that the public sector, what I really like about it is that challenge that you don't get to pick your customers.

Okay.

And a lot of the services we're doing there are ones that people, they don't have a choice to use. The-

Right. You're not choosing who to appeal to.

No.

You're saying, "These people need this service, and we need to provide it."

Exactly. They don't have a choice to say, "Well, I don't like paying my taxes here, so I'm going to go pay them in Finland instead."

Sure. Sure.

Or they do, but it involves more aeroplanes.

Yeah.

So it feels like more of an obligation to build really great services but to build them that everyone can use. You can't kind of segment and just say we're going to focus on our 90% customers.

Yeah, and the stakes are there, right? They're important services that need to be delivered and need to work.

Yeah, very much so. One of the things I've been really fortunate in my career is being able to choose where I want to make an impact. And I think for me, public services, a lot of what you do really does make that impact. You're building things for people that provide them housing or provide them kind of the money they need to live their day-to-day lives or just make their life slightly less frustrating on a typical day, hopefully.

Right. Yes. Okay, so on that note, if we look at the current landscape of public services, how error-free is this whole system of technology and people?

[sighs] I'd love to say error-free. One of the interesting things about working in public sector is you meet a lot of people who are passionate about their work and a lot of people who they could move to the private sector, and they could balance kind of the impact and salary differently. But they-

But they really care about what they're doing-

Yeah, they choose to be there

... is what you're saying, yeah.

Very much so. That said, what we see a lot is systems working just under incredible pressure like the overall system.

Yes.

I was looking at something recently from the MOJ.

Okay, yeah.

Yeah, the court backlog since COVID has more than doubled. I know some of the people working in there, and their teams haven't doubled.

Yes.

So there is this kind of large pressure, large workload, and yeah, in an environment like that, with the best will in the world, mistakes are going to happen.

Yes. Right. Okay. And the reason that I ask that question is because looking forward and looking at new technologies that maybe introduce errors, it's important to look at what's happening currently rather than comparing against an idealized state.

Yeah. [sighs] It's one of those, and I'm sure somebody will have put a name to this, but at the minute, errors happen in lots of kind of small doses. And nobody's name gets attached to that in aggregate. Whereas if you're the person who signs off and says, "We're going to put a new system," and it might have a 98% success rate, and it might be higher than the current one, those 2% errors are still going to have your name attached. And it's still going to be on the pages of the newspapers with your name. So yeah, I try and bring it back to the not is this perfect, but if we're looking to make a change, is this better than what's already there? Does it help rather than harm?

Yeah. And hence can we use that to incrementally work our way forwards-

Inc- yeah

... towards perfect, right? Eventually, I guess.

It's really good incrementally-

Mm-hmm

... because yeah, it's the only way it works, right? If you sit there planning for perfect you're going to be sat there planning for perfect for a very long time. Whereas incrementally, you can still go to bed going, "Is today better than yesterday was?"

Yes.

Yes.

Right. And when we're delivering things at such high stakes, that's actually really important. We don't have the luxury of planning everything out and saying, "Well, we'll have to just live with what we have already." We have to keep getting better.

You mentioned high stakes. It's interesting, isn't it? Well, I think it's interesting, because there are certainly some cases where we just can't tolerate error.

Mm-hmm. Right. Yes.

Places where we have good appeal systems and where kind of issues come up, but they're kind of well corrected and ultimately resolved and maybe kind of in those cases, error in parts of the system is okay. There are other things like prison sentences or parole decisions, or whether we give people a really important benefit where kind of errors can have real and immediate harm, and it's probably we've got that greater obligation to make sure that by the time the decision turns into action, it is the right one. It's that balance, where can you move fast and correct later, versus where do you have a system that needs to be right first time?

Yes, amazing. So looking at taking out just the technology aspect of what we've got already, what have we kind of assumed in the past about how computers work and how a computer system can deliver, I guess, accuracy and getting these decisions right?

Yeah. I definitely assumed that computer systems would be able to take on at least 90% of my admin by now.

[chuckles]

I think that's true. But I think when you think of computer systems, I certainly think of them as these predictable things that are the same today as they were yesterday.

Right. You give it a sum, you say, "What's one plus one?" The answer's two yesterday, it's two today, it's two tomorrow.

It's two tomorrow. And we're kind of hitting this point now with systems where you give it a sum and most of the time it's two... but occasionally it might be three or one. And that can be really disconcerting. Because for the last 50, 60 years, we've been investing authority in computer systems.

Yes.

You asked about error earlier. We know that we make errors because we're human, we make errors ourselves. We come home and go, "Oh, it was at 7:00, not 7:30."

[chuckles] Right.

Or certainly I do. But I think with computers, we've got this kind of, we put them on another level and that's why when they do get it wrong, and I was thinking the Horizon scandal as one kind of really terrible example of this, it can be so harmful because it takes us so much longer to realize that as a possibility. If we now move into a point where actually that's more of a kind of a core aspect of what they are, they're probabilistic instead of deterministic, and I'm thinking about systems with AI, then if we invest that with the same authority, then it's not going to go very well.

Yes. All right. Okay. So building on that, if we think about these critical systems that we're creating, there's so many things now that we can't take for granted. Is there a way to enumerate those things and maybe think through what the consequences are for building things in the future?

I think that enumerating it, as you say, there's definitely things like probabilistic, not deterministic. There's some examples like heavily influenced by context-

Okay. Yes. Right

... rather than just the clear instructions that are given. But I think that actually one of the interesting things is, as you start to list those out, you can see parables to the way we used to... 'Parables'? Parallels - to the way we used to operate, which is how people work.

Yes.

So people aren't deterministic, they're probabilistic. They don't always give the same answer. They're heavily influenced by context. And they can be confused if you give them too much information, similar to the AI systems. They can be misled by a bad actor. If you think the con man going up to a cashier in a casino.

Right.

But we built successful systems that have humans operating within them. I think what it comes to is when you look at it as a critical system, what do you mean by system? What's the boundary of that?

Right.

I used to think of it as kind of a computer system. The box set or laptop, or depending on how old you are, small USB stick that you'd run your operating system on, and that was a system. But there's another sense of it, which is that kind of wider thing, the kind of like a solar system-

Yes

... with all the interrelated parts that kind of combine. And if we think of AI as being one of those interrelated parts, but other parts being humans or audit steps, correction steps-

Yeah

... then you can end up with something where even though each part isn't foolproof and can make errors, the overall system gives you the right result in these kind of critical environments.

And in fact, that's a fairer comparison, isn't it? Because we have no systems that work with one human where it depends on the state of that human in that day and whether they get the answer right in that moment or not, right? We run literally none of our systems like that.

Yeah. And whenever you see it creeping in, you don't normally see calm, ordered results with it.

Right.

Because it's the workload and it's the stress and the like. So, yeah, having the layers, we talk about human in the loop. But it's that combination, how are the two systems working together, the kind of the person and the machine.

Yeah. And I suppose, are there kind of principles that we can pull out of how to responsibly build a system?

There definitely are. So there's the field of AI safety is kind of now moving from that, looking specifically at individual AI bits to kind of taking that step back and looking at an end-to-end system that involves AI, how do we make sure that that overall works?

Yes.

And there's probably three categories of things we can look at there.

Okay. Yeah.

So as much as possible, if we can kind of build things right or architect it right, make the right choices early on then we eliminate the risk from ever arising. You know?

So actually planning, pre-planning.

Yeah, exactly. These systems can, or these parts can be hard to correct in every circumstance later. So why don't you kind of plan it right to start with? So we do things like use case, making sure that we're working to the right use cases, and that's really deliberate. You mentioned principles, kind of making it a deliberate choice of are we answering questions, are we augmenting a human decision or are we automating one? Any one of those for a particular circumstance can be the right choice.

Yes.

But making sure that you're there deliberately and not kind of wandering into one will make sure you put the other surrounding bits around it.

Yes, exactly. Exactly. Like being precise about what you're doing so you can put the right guardrails in.

Very much so.

Yeah.

And you mentioned guardrails. That's another kind of the principles. You can put in place technical guardrails that seek to kind of minimize harm, so make sure that you're not just trusting an AI component with all your data and all your actions.

Right. Yes.

Because there will come a day when it will just nibble away at the rules you've set. So how can you kind of work out, in the same way you would with a new employee, what does this need access to? What does this need to be able to do? And be conservative with that.

Exactly. Yeah.

Yeah. And then probably the most important for me is make sure that it's never the be all and end all, that you've got supervision and appeals. If you've got a system being run with an AI component, make sure that there is a human whose name is attached to it.

Yes.

That means that you've got somebody who is really thinking through the repercussions of this, making sure that full circle, it works for 100% of citizens, say in public sector, and not just the 90% to the cost of the 10. But also that if things aren't working right, that there is a person who can say, "Actually no, this decision isn't the correct one", and can use that thing that we have ultimately, which is discretion to go the other way and to correct the outcome.

Yes. And I wonder if it also perhaps relies on a general level of education at the senior leadership level, so that when there are errors, people are able to look at it and analyze it in the right way. So not just obviously having this one named accountable person - I can see how important that is, but actually making sure that the people around there, the whole governance system has the understanding to be able to hold them to account.

It's really hard because we're in this transition period where there is so much change and people are getting used to so many new concepts. It's kind of the paradigm shift moment. But I think you're 100% right. It's that, how do you start to treat these things as correctable?

Yes.

Rather than as kind of the bulletproof machine, two plus two equals four, forever will. Because that brings in not just different controls, but also a different mindset.

Yes. Right. Exactly. A new way of thinking about it. Well, [chuckles] and on that note-

Yeah

... I'm very aware that, as you said at the start, it's really easy, particularly when looking at these critical government services, to look at how they can go wrong.

Yeah.

And I feel like we've covered that quite extensively. So on the flip side, what are the positives if we get it right? Why is it so important that we do invest the time to properly understand this and look at where it can take us?

I love that because you're right. Why do this at all if there isn't that major upside?

Right.

I think some of it comes from looking at those backlogs and thinking, what would it look like to turn this around? Because we can look at the stats which is a large number of people waiting for their court case, but actually behind each one of those is someone who's just not able to move on with their life.

Right.

They're waiting for a family situation to be resolved, or they're feeling in the right on something, there's that sense of aggrievement. How do they move on? It's a benefits appeal that someone is waiting for. So there's real cost to the delay. I think one real positive is if we can use this to kind of scale our effort and to give people things like justice or answers much quicker.

I think the other thing, though, is looking at people who have maybe struggled with the transition to digital. Quite often, digital is about how do you fit within a process? How do you fit within kind of a box? How do you follow my way of thinking about the world?

Exactly. Here's a form. Yeah. It's your job to fill in this form. Yeah.

Yeah. And if we could turn that around a little, so that you're using AI almost like that personal concierge for each person. We know that humans already do a really great job, citizens advice bureaus, charities, volunteers, in helping people navigate government systems. And it feels like there's a huge opportunity to use AI to provide that kind of personalized service to each person that kind of meets them where they are.

Yeah.

So yeah, I'm really positive about that. I think that will be a great outcome.

Right. Because what you're saying is that the possibilities are doing more and delivering more of these critical services that, as you say, we're already behind on and doing it better. That's where we're heading for.

Yeah, very much so. The reason to move forward is not just to take what we're doing right now and deliver it cheaper.

Yes.

It's to be able to ultimately deliver something that feels really different, that uses information it already knows about you to take work off your plate.

Mm.

No one gets up in the morning thinking, "Today I want to fill in a parking permit application."

No.

They think, "I want to have mobility."

Yes.

And to do that, I need a car somewhere near my house. And to do that, I need to fill in this form. And it's a step, step, step. But what if we can use it to eliminate some of those steps and just go back to giving them the mobility or giving them the access to healthcare or education that they're looking for, without some of that life admin.

Right. Or actual barriers, even worse actual barriers.

Yeah.

Yeah.

It doesn't feel that far away that we'll be getting that. We know that kind of within the centre of government, they're already working on what their kind of agentic experiences look like. It's a technical term we haven't used yet today, but fundamentally, what does it mean to have systems that don't just respond to your immediate requests, but try and predict them and facilitate them and proactively offer you solutions

Yeah, I was going to say, to use a human term, like 'think around it', essentially what you're asking.

Think around it.

Yeah.

The kind of thoughtful person who says, "Hey, I don't know if it's useful, but..."

Mm-hmm. Right. Exactly.

So it feels like that starts to open up possibility. I think the other thing is if we don't move ahead, say in public sector to do that, it feels likely that third parties will come in to look to fill that void. And then kind of people's relationships with their government will be intermediated, which from a point of view of trust and from a point of view of a really positive governmental system that with engaged citizens feels worrying.

Right. It's complex. There are good possibilities... but also less good possibilities.

[chuckles] Definitely. I think 'it's complex' is the perfect description that applies to this.

[laughs] It's my answer to all technology questions. Yeah.

Yeah. But no, I'm really excited. I think that there are risks, and I hear people talk about the risks, but the way we reap the benefit is by having that discussion, not from a there are risks, so we can't but more there are risks, therefore we need to.

Yes, exactly. And surfacing the risks is the important part of how you deal with them, right? If you don't know about it, you're not going to be able to address it.

Yeah.

Okay, brilliant. So just finally, quickly I've been very heartened by the progress we've seen over the last 20 years in terms of government services being delivered digitally and how that's definitely affected my interactions with the government in a super positive way. What things do you think are on the horizon that maybe the public can expect to see over the next five years, the next 10 years?

Um...

I'm not going to hold you to it, but just-

Okay. [chuckles]

... you know.

Yeah, I will come back in five years' time and listen to this podcast and go, "Oh, wow-

[laughs]

... you were spot on or not."

I think it's going to be a really interesting time. Britain was, for a long time, viewed as very much the lead in the world on how we offer digital services. We were first or second in the world rankings on that, and that's thanks to a lot of hard work from the people in GDS that then spread out to the departments. Over the last few years, kind of as we've dealt with some of the thornier problems, we've slipped down in the rankings a little. Still in the top 10, but slightly lower. It feels like over the next few years, there's a real hunger and appetite and the right people in the right roles to kind of make that happen again, and to say, with this new technology coming through, how do we leapfrog back to providing one of the world's best services?

That's likely to come in a few areas. In the UK, because we're a large country, 70 million and because we have really strong rules on privacy and as a vibrant democracy, some things are harder. So sharing data across government has always been one of those things that can unlock so much but it's just always been that kind of five years' time thing, a bit like fusion.

Yeah.

It feels like in the next five years, that really will become a reality. One of the things that AI can do is help kind of map the data across departments in a way that would've taken a lot of manual effort in the past. And because there's such compelling use cases for when you get the data right for AI, it also feels like the funding and political will is there.

Yes. Right.

What that means for you or me is that when we go to apply for a new driver's license, our photo comes across from our passport, our address data can be suggested from something we did for health recently or for the council. But that should all be done in a way that's really kind of respectful of your privacy and your preferences.

And secure. Yeah.

And secure, definitely.

So I think there's a lot of positive opportunity come in for how we join up government services. And then in terms of your experience of that, there's a GovUK app, and that's about kind of bringing everything together into just that one place that then switches from being reactive to proactive. So it's a notification of-

Great

... "Hey, you need to update your passport" with the AI options, "You need to update your passport... Check this is correct and hit go, and we'll do it for you."

Yeah.

So how can we broaden people's awareness of what they're entitled to across government, get the data on them more accurate so that there's less work for them, and ultimately kind of reverse the trend in kind of slightly decline in trust for government services and kind of enjoyment of to get back to kind of top of the rankings for delivering a really positive experience for people?

Yeah.

That for me, I think next five years. Next 10 years? Flying cars and zero rate tax.

Yeah. [laughs]

Not sure.

Aw, thanks so much, Alex. Thanks for sharing your experiences and helping us to shine a light for others.

Oh, it's been really nice. Thanks, Zoe, for having me.

This Digital Lighthouse episode was edited by Steve Folland and produced by Patrick Anderson. The theme music was written and recorded by Ben Baylow. A huge thanks to our sponsor, Softwire, for their continuing support right from the inception of the show in 2019 to the present day.

If you love the podcast, please let us know with a rating and review on your platform of choice. We're always looking for feedback to ensure we're making the best show possible. And if you'd like to take part, please drop us a line at thedigitallighthouse@softwire.com.

You've been listening to The Digital Lighthouse with me, Zoe Cunningham. Thank you for sharing your time with us and stay safe on this wild technological ride that we're all on.