Human in the Loop Has to Mean Something

Angela Lipps spent more than five months in jail for crimes she said from the beginning she could not have committed. She lived in Tennessee. The crimes happened in North Dakota. She said she had never even been to North Dakota. That seems like the sort of detail somebody might want to check. Instead, facial-recognition technology had produced Lipps as a possible match during an investigation into bank fraud. That lead made its way to Fargo investigators. A detective compared photographs and continued pursuing her as a suspect.

In July 2025, U.S. Marshals arrested Lipps in Tennessee. She remained incarcerated for more than five months. Her bank records eventually provided something substantially more useful than an algorithmic resemblance: evidence placing her in Tennessee while the North Dakota crimes were occurring. The charges were dismissed in December 2025. Lipps filed a federal civil-rights lawsuit on September 15, 2026, against the City of Fargo and former detective Lucas Heck. She alleges that during her incarceration she lost her home, car, belongings and dog and suffered additional harm, including being deprived of medication and access to her dentures.

Those allegations now have to work their way through the courts. But the underlying investigative failure isn’t merely an allegation from Lipps. then-Fargo Police Chief Dave Zibolski publicly acknowledged errors in the investigation, and the department changed its facial-recognition procedures afterward. The case has been covered by MPR News, and the federal lawsuit itself is publicly docketed as Lipps v. City of Fargo et al.

And there is one fact about this case that should make everyone building consequential AI systems uncomfortable: There was a human in the loop. It didn’t save Angela Lipps.

“Human in the loop” sounds better than it is

Human in the loop has become one of those wonderfully reassuring phrases we use when discussing AI. Don’t worry. The machine isn’t really making the decision. A human reviews it. Great. What does “reviews it” mean? Because a human who looks at an algorithm’s answer and says, “Yeah, that seems right,” is technically a human in the loop. A human clicking APPROVE hundreds of times per day is a human in the loop.

A human who receives an automated recommendation but lacks the information, time or authority to challenge it is a human in the loop. A human who assumes the computer performed analysis that it never actually performed is also a human in the loop. None of those things constitute meaningful oversight. The question isn’t whether a human appeared somewhere in the workflow. The question is: Could that human independently determine that the machine was wrong?

And then there is an equally important second question: Did the system actually require them to do so? Those are very different standards.

A lead is not reality

Facial recognition is probabilistic. It doesn’t look at a photograph and discover some metaphysical truth about the identity of the person pictured. It compares information and produces potential matches according to a model. That can be extremely useful. If investigators have an image of an unknown person and software can search an enormous collection of images and say, “Here are some people you might want to investigate,” it can dramatically reduce the search space.

But that output is a lead. It is not proof that the person committed a crime. It isn’t even necessarily proof that the person in the photograph is that person. That distinction seems painfully obvious when written down. Apparently it isn’t obvious enough. The ACLU’s tracking of wrongful facial-recognition arrests documented at least 14 publicly known wrongful arrests associated with police reliance on erroneous facial-recognition results as of April 2026. Lipps therefore isn’t useful as an example because she represents one bizarre, impossible-to-repeat failure.

She is useful because she doesn’t. This has happened before. Different people. Different police departments. Different circumstances. The same dangerous transition from the computer suggested this person to let’s investigate this person as though the suggestion itself carries evidentiary weight. Once a machine produces a name, that name can become the center of the investigation. Now the questions subtly change. Does this photograph look like Angela Lipps?

Could this identification belong to Angela Lipps? Does this evidence fit Angela Lipps? But the original question was: Who committed the crime? That’s not semantics. That’s architecture.

The dangerous AI answer is usually the plausible one

People talk a lot about AI hallucinations because they can be funny. Ask a model something and occasionally it will confidently produce absolute nonsense. Those failures are easy. A person notices. The dangerous answer is the one that is plausible and wrong. If facial recognition identifies a 94-year-old Norwegian man as the twenty-something woman visible in surveillance footage, somebody is probably going to notice. If it returns someone who looks sufficiently similar to a blurry image, the answer feels reasonable.

Reasonable answers are precisely where independent verification becomes important. Humans have a tendency to trust systems that appear sophisticated. Put percentages next to an answer and it feels scientific. Put “AI” in the product description and somehow we occasionally become reluctant to ask whether the computer actually knows what the hell it is talking about. That is automation bias, and simply putting a human downstream from the machine does not make it disappear.

It can make it worse. The machine proposes an answer. The human begins evaluating the proposed answer rather than independently evaluating the problem. Now the computer hasn’t technically made the decision. It has merely framed every decision that follows.

Meaningful human oversight has requirements

Meaningful oversight requires more than a person and a button. The human needs enough information to independently evaluate the machine’s conclusion. They need to understand what the system actually did, and just as importantly, what it did not do. They need enough time to investigate contradictions. They need authority to reject the result. The system needs to tolerate that rejection, and the organization surrounding that human needs to expect disagreement rather than treating it as inefficiency.

If any of those pieces disappear, the human gradually stops being a decision-maker. They become an approval mechanism. That is why the Lipps case is so striking. Before depriving somebody of her freedom, the investigation needed to survive an extraordinarily basic question: Was she even there? Her bank records reportedly helped establish that she was more than a thousand miles away. That is not sophisticated counter-AI technology.

It’s a bank record. Sometimes the appropriate response to artificial intelligence is regular intelligence.

This isn’t only about facial recognition

The same architectural problem is appearing everywhere. AI can recommend whether someone receives credit. It can flag financial transactions as fraudulent. It can screen employment applications. It can prioritize patients. It can identify people from cameras. It can determine what information a human operator sees first. And increasingly, AI can participate in military systems. The consequences are different. The ethical principle isn’t. Adding a human somewhere downstream does not automatically make an automated decision safe or ethical.

If the human cannot realistically challenge the machine, the machine is making the decision. If the human doesn’t receive independent evidence, the machine is framing the decision. If the human is punished for disagreeing with the machine, the machine effectively has authority. And if a human has three seconds to evaluate a recommendation produced by an enormous computational system, calling that “human oversight” is theater.

When a human life is at stake, theater isn’t good enough. This is where I have a hard ethical boundary. I will not work on autonomous AI systems that independently decide that a human being should be killed. That decision must remain meaningfully human. Not technically human. Not “there is an operator somewhere watching a screen” human. Meaningfully human. The person making that decision needs information.

They need time. They need authority. They need the ability to say no. And the system must be designed to accept no as an answer.

Ethics should change the architecture

If communication fails, the system should fail safely. If the model’s output conflicts with independently established facts, the facts should win. Those are engineering decisions. That’s what makes the ethical principle real. Otherwise “responsible AI” becomes another phrase somebody puts on a slide deck between “innovation” and “leveraging synergies.”

The computer did not arrest Angela Lipps

Humans did. That distinction matters enormously. It would be easy to look at her case and conclude that the problem was bad facial-recognition technology. That’s too easy. Technology will improve. Models will become more accurate. False-positive rates will decline. None of that eliminates this problem. No sufficiently consequential system should be designed around the assumption that its model will never be wrong. The deeper failure occurs when a probabilistic output acquires authority it should never have had.

Angela Lipps lost more than five months of her freedom before the system corrected itself. Whatever ultimately happens with her lawsuit, that should make every engineer building consequential AI systems uncomfortable. It certainly makes me uncomfortable. So the next time someone tells me an AI system is safe because there’s a human in the loop, I have a question. Could that human have stopped what happened to Angela Lipps?

If the answer is no, the human isn’t meaningfully in the loop. They’re just standing next to it.

Sources

The underlying Lipps case and Fargo Police Department response are covered by MPR News — “Angela Lipps sues Fargo over AI-aided arrest”. The broader record of documented wrongful arrests associated with facial-recognition results is maintained by the American Civil Liberties Union.