Seen But Not Known: What Happens When Facial Recognition Gets it Wrong
Last time, we talked about bias in AI in general terms, and what happens when a computer learns from a world that is not fair to begin with. This piece looks at one very specific place that shows up: facial recognition software.
When this technology gets it wrong, the consequences are not abstract. They are a man handcuffed in his driveway in front of his children. They are a grandmother looking after her grandchildren who ends up in a police interview room. They are a teenager whose bag of crisps gets flagged by a scanner as a possible weapon.
Facial recognition uses AI to scan faces in real time and match them against a list of people that police, security services or law enforcement are looking for. However, it has a troubling track record, and the rules meant to control it are still being written while the cameras are already switched on.
What happened to Robert Williams
Robert Williams is one of the clearest examples. In January 2020, police in Detroit arrested him outside his own home, in front of his wife and two young daughters, and held him for thirty hours. A facial recognition system had matched him to a man who stole watches from a shop. When police showed him the security footage, he said, "That's not me." He was right. Police later admitted their own investigation had been sloppy. Williams had a clear alibi, a video he had livestreamed of his commute that placed him nowhere near the shop, and nobody had checked it before arresting him. He was not even the best match the system produced. He was the ninth best.
His case was the first time a wrongful arrest caused by facial recognition was properly documented in the United States, and it was not the last. Every single known wrongful arrest of this kind shares the same root cause: proper checks were not carried out before acting on the computer's answer. One man in Louisiana spent a week in jail despite being forty pounds lighter than the person in the footage. A woman eight months pregnant in Detroit was held for eleven hours after being wrongly matched to a carjacking. In the UK, a man was wrongly accused of a crime in a city he had never even visited, because scanning software confused him with someone who simply looked similar.
You might recall from our last piece that Joy Buolamwini and Timnit Gebru’s Gender Shades study of 2020 found facial analysis error rates of 34.7% for darker skinner women compared to 0.8% for lighter skinned men. Given how fast this technology develops you might be tempted to think, well that was like a bazillion years ago in AI tech terms. However, even in 2026 researchers from the University of Cambridge found that Essex police's facial recognition system was still far more likely to wrongly identify Black people than anyone else. Four out of the six people wrongly flagged in their study were Black, even though Black people made up only a quarter of everyone tested. Essex Police stopped using the technology as a result.
The UK’s Information Commissioner's Office, the watchdog that checks how organisations use our personal data, has since properly audited that force, and Leicestershire Police too, finding things done well alongside things that need fixing, with follow up now underway. The watchdog's own research found that the public's biggest worries are:
Whether the technology is accurate,
Whether officers are properly trained, and
Whether there are proper safeguards against bias.
That seems like a fair list of demands to me.
None of this is a coincidence. These systems learn from photo collections that were never representative of everyone in the first place, built and checked by teams that were not diverse enough to notice the gaps. The people most likely to be wrongly flagged are the same people already most likely to be stopped and questioned by police, and so the intuitional racism reported in studies like The Macpherson Report (1999), The Casey Review (2023) and the HR Rewired Independent Review (2025) are compounding the bias already baked into AI outputs when humans then make decisions on them. The technology does not invent that unfairness. It repeats it, at scale, and dresses it up as objective science, but ironically it is often human discrimination which then fails to apply the judgment needed to prevent harm being caused.
Some Genuine Progress
In 2024, Detroit Police were forced, as part of a legal settlement in the Williams case, to back up any facial recognition match with real, independent evidence before making an arrest, and to review every case since 2017 where the technology helped get a warrant. That is a real win, and it came from one man refusing to accept what happened to him as simply the price of new technology.
The UK has also had its rules tested in court, twice now. Back in 2020, judges ruled that South Wales Police's use of the technology broke the law. That ruling shaped how every police force has written its policies since, including the Metropolitan Police's rules, updated in 2024. Those rules were challenged again this year by Shaun Thompson, a community worker who was wrongly identified by Met cameras and nearly arrested before he managed to prove who he was, alongside campaigner Silkie Carlo. In April 2026, the High Court sided with the police, ruling that the Met's policy was clear enough to be lawful. Thompson is appealing. It is worth being clear about what that actually means though. The court said the police's rulebook is written clearly enough. It did not say the technology works properly, or that people will stop being wrongly identified. Those are two very different questions, and it is easy to mix them up.
Last year the government ran a proper public consultation, admitting the current rules are a confusing mess spread across several different laws, and it now wants one clear law instead, with a dedicated watchdog. The person who currently holds that watchdog role went further still, suggesting that the communities most affected by this technology, which tends to mean Black and Asian communities, should have a genuine say before decisions are made, not just a survey afterwards. That is exactly the kind of thing campaigners have been asking for, and it is encouraging to see it land in an official government document.
Other countries are ahead of the UK. The European Union has banned live facial recognition scanning by police in public places altogether, with further protections coming in August this year. In America, more states keep adding protections even while a national law goes nowhere. Maryland now makes sure people are told if facial recognition was used against them. Virginia brought in a law this July requiring police to get a warrant first. Milwaukee banned police use of the technology completely after residents pushed hard for it. None of this is perfect, some police forces in banned cities have simply asked neighbouring forces to run searches for them instead, but the direction of travel is real.
The cameras did not wait
At the same time as all of this, the number of cameras keeps growing. The UK government has funded forty facial recognition vans, four times what it had before, plus a new national centre for AI in policing. Fixed cameras are going up across London's West End. Officers are increasingly able to scan a face straight from a handheld device, and this is being tested with body cameras and even drones at protests.
Shops are moving just as fast. One system used across UK retailers logged over half a million alerts last year and is about to start telling police automatically, within seconds, when it thinks it has spotted someone. The evidence still shows Black and Asian shoppers are more likely to be wrongly flagged, and people wrongly caught up in it often still cannot find out why, or how to challenge it.
What you can do
Contact your local elected representatives or civil liberties oversight boards. Demand to know what legal frameworks govern the deployment of live biometric surveillance in your area, and what dynamic safeguards exist to protect citizens against algorithmic misidentification.
Support international digital rights organizations like the Electronic Frontier Foundation, Access Now, European Digital Rights, Liberty, Big Brother Watch and the Good Law Project, The Algorithmic Justice League, Amnesty International, Human Rights Watch, Asociación por los Derechos Civiles, Coding Rights, Internet Freedom Foundation, Digital Rights Watch, The Molly Rose Foundation, Collaboration on International ICT Policy for East and Southern Africa, Spaces for Change, Right2Know Campaign, Digital Rights Foundation, South Asian Human Rights Association or local human rights defenders. These groups are leading the global front lines, challenging biometric surveillance, tracking predatory tech procurement, and fighting algorithmic bias in courts and parliaments worldwide.
Exercise your right to information. If you see a mobile surveillance van or public-facing biometric camera, question its legal basis, data retention policies, and operational scope.
The expansion of this technology is not happening in secret. It is happening in plain sight, quickly, while the people meant to be governing it are still catching up. The most useful thing any of us can do is pay attention, because most people still have no idea how much of this is already around them.
We’re also happy that Diverse AI is working to challenge some of these issues with facial recognition. We aim to further socio-technical understandings of how to democratise the purpose, use and access to AI and data driven technologies Our research projects are inter-disciplinary and collaborative and aim to create breakthroughs in new AI algorithms, supporting architectures, processes, input datasets, evaluation and governance. One of our projects for instance aims to address the lack of representation in data sets by creating a first-of-its-kind diverse and inclusive image dataset containing the images of cultures typically absent online, by reflecting the cultural, geographical, and demographic characteristics of these groups. The research will also highlight underrepresented groups such as LGBTQ+, disabled / neurodiverse, female, religious, and age groups among these communities and contribute to reducing some of the issues associated with live facial recognition if embedded in criminal justice systems.
Next time, we turn to agentic AI, where AI stops just watching and starts acting on its own. It raises a whole new set of questions, and I think you will find it just as important as everything we have covered here.
