Your Database Is Not Probable Cause

I have spent a large part of my career building systems that answer questions. Databases are very good at this. Give me a sufficiently well-designed dataset and I can tell you which customers match a condition, which servers missed a patch, which transactions look unusual, or which vehicle was observed near a location at a particular time.

There is a dangerous little linguistic jump that happens when these systems move outside engineering: match quietly becomes true.

Those are not synonyms.

Suppose I write a query looking for a black Dodge Durango seen within a geographic area during a particular time window. The database returns three vehicles. What has the database established? Exactly one thing: three records satisfied the conditions of my query. It has not established that one of the drivers committed a crime. It has not established that my search parameters were correct. It has not established that the underlying observations were complete. It has not even established that the vehicle classification was perfect.

In software engineering this is mundane. We spend enormous amounts of time asking whether the assumptions behind a query are correct. In higher-stakes environments, somehow the output can acquire an authority the underlying computation never earned.

SELECT is not a verdict

A database query is an implementation of a question. If the question is wrong, the database may give you an exquisitely precise wrong answer.

This gets more complicated when machine learning is involved because the query itself may contain probabilistic classifications. A camera may estimate make, model or color. A facial-recognition system may return similarity candidates. A fraud model may assign a score. Those are useful signals, but each introduces uncertainty that should travel with the result.

Instead, interfaces have a habit of sanding uncertainty away. The user sees a photograph, a name, a vehicle or a big red warning. The complicated statistical machinery disappears behind a clean screen, and clean screens feel authoritative.

Good interface design can accidentally become bad epistemology.

Authority has to be designed

When I design a system, one of the questions I care about is not simply what the software can calculate, but what authority that calculation is allowed to have. A recommendation can populate a queue. A score can trigger additional review. A match can surface evidence. None of those necessarily should be permitted to make the consequential decision themselves.

This is why “human in the loop” is not enough as a label. If the human receives only the machine’s candidate and none of the contradictory evidence, the human is not independently reviewing anything. If the workflow makes rejection difficult, the human is functioning as an approval button. If people are expected to process hundreds of alerts per day, the organization has designed automation bias into the job description.

The human needs enough information, time and authority to say the machine is wrong.

Build systems that can lose arguments

I think this is one of the most important design principles for consequential AI: the system has to be capable of losing an argument with reality.

If physical evidence contradicts a model, physical evidence wins. If authenticated records contradict a similarity score, the records win. If the system cannot establish enough confidence, it should be able to say “I don’t know” without somebody treating that as a software defect.

That sounds obvious until incentives enter the room. Organizations buy expensive systems because they want results. Vendors want impressive demonstrations. Managers want efficiency. Users learn that the fastest way through a queue is accepting the recommendation.

Suddenly uncertainty becomes friction to be engineered away.

Some friction is there for a reason.

A database can help you find the needle in the haystack. AI can make the haystack searchable in ways that would have looked like science fiction twenty years ago. I am enthusiastically in favor of using those capabilities.

But when the consequence is somebody’s job, money, liberty or life, we have to preserve the distinction between finding a candidate and establishing a fact.

Your database can tell you what matched.

It cannot tell you what you are morally entitled to do about it.