The most dangerous sentence in a software review is not “I don’t know.” It is “The AI said this is secure.”

“I don’t know” is useful. It identifies a boundary and creates an opportunity to investigate. Unearned certainty does the opposite. It closes the discussion before anyone has examined the risk.

AI coding assistants can be remarkably effective at translating requirements into working interfaces and services. That creates a new challenge for organizations: people who have never designed a trust boundary can now build systems that appear to contain one.

The resulting code may be tidy. It may have tests. It may even include security terminology in comments. None of those things proves that the author—or the model—understood the threat.

The questions that never get asked

Imagine an employee creates a customer portal. The user signs in, sees a dashboard, and downloads an invoice. Everything works in the demo.

Did anyone test whether changing an invoice ID exposes another customer’s document? Whether a user can modify their role in a request? Whether the application trusts an organization ID supplied by the browser? Whether a password-reset token expires and can be used only once?

Those are not exotic attack techniques. They are ordinary questions about authorization, ownership, and state. But you have to know to ask them.

Now add a third-party package, a cloud storage bucket, a background job, and an AI-generated integration that quietly logs request bodies. The risk expands beyond any one file of code.

A model can answer a question it receives. It cannot reliably guarantee that the human operator asked every important question.

Expertise is recognizing missing context

In experienced engineering teams, review is not simply a hunt for syntax errors. It is a conversation about assumptions.

Why does this service need that permission? What happens when two requests arrive simultaneously? Which system is authoritative? Can the client forge this value? What is the rollback plan? Where does sensitive information leave our control?

These questions come from accumulated experience with failures, production incidents, and systems that behaved differently under real load than they did on a developer’s laptop.

That experience is precisely why eliminating junior engineering opportunities is shortsighted. We need people learning to ask better questions, not merely producing more code with fewer keystrokes.

And the distinction has nothing to do with a degree. A curious self-taught developer who actively seeks review may be safer than a credentialed engineer who refuses it.

Design for uncertainty

Organizations should make uncertainty a normal part of the development process. Require developers and AI-assisted builders to document what the system assumes, what data it handles, and what security properties have not been verified.

Create a simple escalation path: applications handling credentials, customer data, payments, or external integrations receive qualified engineering review before production access. Provide approved components for authentication, storage, and logging so teams are not reinventing sensitive mechanisms.

Treat AI-generated tests as a starting point, not a certificate. Include adversarial scenarios and independent review. Make it easy to say, “I need help understanding this.”

The goal is not to embarrass people who are learning. It is to prevent confidence from outrunning competence.

The best engineering cultures reward the person who finds an uncomfortable question before an attacker finds the answer.


Discover more from Christine Alifrangis

Subscribe to get the latest posts sent to your email.