Imagine that you sit down with Claude Code or Cursor or Copilot to write a careful prompt:
Build an API that is multi-tenant. Every query must be scoped to the tenant ID grabbed from the JWT. Never, ever trust a tenant ID that comes in on the request body or query string.
You watch it scaffold the first few endpoints. It does exactly what you asked. The tenant ID comes off the JWT claims and gets passed into your repository calls. Feeling confident, you go get some coffee and come back fifteen minutes later. It’s on endpoint number eleven now, and you watch it reading tenantId straight out of the query string.
“What in the world?!” You didn’t tell it to stop using the JWT. Is it stupid, or is it trying to sabotage you?
AI: The Dumbest Genius
If you tell AI to create a to-do app and sit back for a few minutes, it can create a flawless one 10 times as fast as you could have even typed in the code for one yourself. Enough experiences like this and you get lulled into a false sense of security. You start to feel like your AI assistant knows all.
Imagine that you hired a developer, told them “the single most important rule on this project is tenant isolation,” walked out of the room, and came back a minute later to find them writing an endpoint that lets any customer read any other customer’s data. You’d assume something was wrong with them. For all our flaws, no human is going to forget a critical thing in just sixty seconds. And because we like to anthropomorphize our LLMs, it catches us by surprise when they do something this stupid.
An LLM isn’t remembering your requirements. It’s predicting the next token, and your requirements are just one of many inputs that are tugging on that prediction.
When your instruction was thirty seconds ago, it still has a lot of pull over what your LLM does. Thirty minutes later, buried in the context under file contents and tool output and the LLM’s own chatter, it has much less.
By the time the LLM gets to implementing endpoint number eleven, it asks itself “what usually comes next here?”, and the answer from the millions of examples it saw in training is “a tenantId parameter from the query string.” Your one critical instruction is fading into the background.
Where I’ve Watched It Forget
Below are three places just recently where I gave the model a clear rule up front and it drifted anyway:
Security
I asked for a “secure” app and got one…mostly. It built out parameterized queries, used input validation on the DTOs, everything you would want! But then I ran a security scan after it was done (something like Semgrep, Snyk, or GitHub’s CodeQL) and it flagged a raw FromSqlRaw with a string-interpolated WHERE clause. (For more on scanning AI-written code, see our post Locking Down AI: Strategies for Uncovering Vulnerabilities.)
Architecture
On a modernization project I told the model up front to use clean architecture, and that the Domain project should have no reference to EF Core because all persistence goes through repository interfaces. It set up the projects perfectly and wrote a nice little README explaining the rule, but about halfway through implementing the feature slices, a handler in the Application layer started newing up a DbContext directly because that was, frankly, the shortest path to making the test pass.
Coding standards
“Use our Result<T> type for failures, don’t throw exceptions for expected error cases.” It followed the rule for the first six methods. Method seven, however, throws an InvalidOperationException. Method eight is back to Result<T>. It’s not even consistent about being wrong!
If this is how these LLMs tend to work, then how do we write quality code with them? Better prompting isn’t the fix, and neither is ALL CAPS. The answer is check gates.
Requirements First, Check Gates After
Here’s the mental model that finally got me out of the “why do I have to keep saying this” loop:
If something is important enough to tell the AI up front, it’s important enough to check for afterward. The instruction is a requirement. The check is a separate gate that comes after.
On my projects, that looks like:
1. Put the rules where they get re-read
Keep a CLAUDE.md or AGENTS.md at the repo root with your most important rules. These get pulled back into context every time you start a new session. Inside a long session it can still drift, but this resets what’s critical every time you start fresh.
2. Turn the rules into tests
An architecture test (ArchUnitNET, or NetArchTest if you’re on the older stuff) that asserts the Domain project doesn’t reference Microsoft.EntityFrameworkCore does more for you than a paragraph of prose about clean architecture. A Roslyn analyzer that flags FromSqlRaw is better than the word “secure” in a prompt. If the model can break the rule silently, it sometimes will. If breaking the rule turns the build red, the model will fix it.
3. Run scans
Security scan, dependency audit, whatever your equivalent is. Put it in your CI/CD process or tell the agent to run it itself and fix what it finds. In my experience, it’s very good at fixing findings you feed to it.
4. Have a second model do a code review
Ask a fresh session (or a different model entirely, Codex reviewing Claude Code’s output or vice versa) to review the diff against the original requirements. Give it the rules and the changes and nothing else. It has no memory of the shortcuts the first model took and no investment in them, and it catches things the original session would swear it didn’t do. It’s code review by someone who wasn’t the developer, which is the whole point of code review.
None of those are new ideas. We’ve had CI gates and code review and static analysis for over twenty years. What’s new is realizing you need to apply them to LLMs as well as humans. And as I wrote in Developing with AI Exposes Your Bottlenecks, the gates are where the work piles up once coding gets fast, so they’re worth investing in.
Where the Analogy Breaks
I’ve been talking about LLMs as a forgetful genius developer, and that’s a useful analogy, but not perfect. After all, a human who forgets something will usually feel bad enough about it to not do it again. Your LLM might not make the same mistake in the same session after you correct it, but then it’ll happily do it again tomorrow with another “Good catch!” when you point it out. It doesn’t learn like that, so you need to set up your processes to catch its weaknesses.
If you want help building a solid process around your AI-assisted development, reach out to us at Trailhead. We have a lot of experience building gates into our projects, and we’d be happy to help you tighten up your AI development processes.
