Essay
The security holes AI coding tools leave behind
I spent about a decade breaking applications before I built one with AI assistance. The vulnerabilities that show up are not new or exotic. They are the same ones as always, arriving faster and looking more finished.
The thing nobody tells you about AI-assisted development is that the code does not look wrong. It looks like the code in the tutorial. It is well-formatted, it has reasonable variable names, it handles the happy path, and it frequently has a comment explaining what it does. It passes the vibe check that a tired reviewer applies at 6pm.
Which is precisely the problem. The failure mode of a junior developer is code that looks uncertain. The failure mode of a model is code that looks finished. Reviewers calibrate on appearance, and these tools have broken the correlation between how confident code looks and how correct it is.
None of the holes below are novel. That is the point. They are the OWASP top ten with better formatting. What has changed is the rate at which they appear and how plausible they look on the way in.
One
Authorization is checked where it is convenient
Ask for an endpoint that returns a project, and you get an endpoint that returns a project. It will probably check that you are logged in. It will very often not check that this project is yours, because nothing in the request looks suspicious: the client sent a project ID, the handler looked it up, everyone is authenticated.
This is the oldest bug in web applications and it is still the one I would bet on finding. It is worth being precise about why the model gets it wrong: authentication is a property of the request, and authorization is a property of the relationship between the requester and the resource. The second one is not visible in the code you are looking at, so a system that predicts plausible code will skip it.
The defense is structural, not vigilant. There is one function, and project-scoped routes go through it rather than each deciding for themselves:
export async function requireProjectAccess({ projectId, auth, requestHeaders }) {
// Check access BEFORE existence, so a 403 vs 404 cannot be used
// to probe which project IDs are real.
if (auth.kind === 'user') {
if (!canAccessProject(auth.user, projectId, requestHeaders ?? new Headers())) {
throw new ProjectAccessError('Access denied', 403)
}
}
const project = await projectServiceAdmin.getById(projectId)
if (!project) throw new ProjectAccessError('Project not found', 404)
return project
}Note the ordering, because it is the part a model will never write for you unless you ask. Authorization runs before the lookup. If it ran after, then a project you cannot see returns 403 and a project that does not exist returns 404, and now the difference between those two responses is an oracle that lets anyone enumerate valid IDs. That costs nothing to prevent and is nearly impossible to notice in review.
Two
Secrets end up where the user can read them
You need to post a form submission to Slack. The direct path is a fetch from the browser to the webhook URL. It works on the first try, which is exactly why it survives.
And now the webhook is in the JavaScript bundle. It is in the git history. It is in the CDN cache. Anyone who opens devtools can post to your company Slack indefinitely, and rotating it means a deploy, and nobody will notice for a year because nothing is broken.
The same pattern produces API keys in NEXT_PUBLIC_ variables, database credentials in client components, and admin tokens in mobile apps. Every one of them starts as the shortest path from a working prototype to a working feature.
Three
Identity is taken from a header
In a multi-service system, some service needs to know who the caller is. The path of least resistance is to put it in a header. The gateway sets x-user-id, downstream services read x-user-id, and it works perfectly.
Until someone sends that header themselves. Now they are whoever they say they are, and the entire authentication system is decorative.
The correct shape is that identity must be unforgeable and downstream services must verify rather than trust. My gateway does two things about this. First, it strips the nine headers it considers spoofable off every request before a handler can see one, so a client cannot smuggle one in. Then it mints a signed assertion that downstream handlers verify:
export function createAssertion(
claims: MiddlewareAuthContext,
maxAgeSeconds: number = 60,
): string {
const currentKey = getCurrentKey() // rotatable, min 256-bit
const now = Math.floor(Date.now() / 1000)
const payload = { ...middlewareClaimsToPayload(claims), iat: now, exp: now + maxAgeSeconds }
...
return `${data}.${sign(data, currentKey.key)}`
}Sixty seconds is not a guess. The assertion is created in middleware and consumed by a route handler in the same request, so a minute is already generous, and it means a leaked assertion is worthless almost immediately. Verification uses a constant-time comparison, because a signature check that returns early on the first wrong byte is a signature check you can brute-force one byte at a time.
No model will volunteer any of that. It will happily write you a working HMAC, and it will write it with ===.
Four
The model becomes an untrusted input
This one is genuinely new, and it is the one I see people get wrong in a way that has no analogue in the pre-AI world.
When an application feeds a model something a user wrote, or an email it received, or the text of a web page it fetched, the model's output is no longer trusted data. It is a transformation of hostile input. If that output then drives an action, you have handed control of your application to whoever wrote the input, and you have done so through a component whose whole purpose is to follow instructions.
Old habits apply directly here and most people are not applying them. The model is a parser of untrusted input, so its output gets validated against a schema before anything touches it. It never emits a query, a command, or a path. And the actions it can take are enumerated in advance rather than inferred from what it asked for.
What I actually think
I am not making an argument against these tools. I built a system of roughly forty services with them, alone, in a year, and I would not have attempted it otherwise.
The argument is narrower. These tools remove the friction that used to be doing security work for you by accident. Writing an authorization check by hand made you think about who was allowed to do what. Wiring up a webhook by hand made you notice you were putting a secret somewhere. That friction was never the point, but it was load-bearing, and it is gone now.
So the thing that has to replace it is knowing what to ask for. That is the actual skill, and it is why a decade of breaking software turns out to be the best possible preparation for building it this way. Not because I write more secure code by hand. Because I know which questions to ask, and I ask them before the code exists rather than after it ships.
The companion piece, on keeping a human accountable for what these systems do, is here.