When an LLM writes code, it can produce something that looks right and even runs in a narrow test but is wrong in structure: wrong API, wrong assumption about state, or a path that never gets tested. The model doesn’t “see” the full codebase or the spec; it predicts the next token. So it can add a happy path and forget the error path, or introduce two flags that can both be true in a bad combination, or leave a resource open. Those are structural bugs — bugs in the shape of the program — not simple typos. Tests that only cover the happy path won’t catch them.
Why the model writes bugs it cannot see: it has no formal model of the system. It doesn’t know “every state must have an error transition” or “this lock must be released.” It only knows statistical patterns from training code. So it tends to add and rarely to delete or refactor. It fills in the obvious next step and often misses the edge case or the cleanup. That’s the additive trap in code form.
Mitigations: use the model for drafts and then review. Run static analysis, linters, and tests that cover failure paths. In critical areas, keep the model on a short leash: generate small patches, run tests after each, and require human sign-off for structural changes. Some teams use formal specs or state-machine descriptions and then ask the model to implement against them — the spec is the source of truth, the model is the implementer.
Hallucination in code is a special case of “confident and wrong”: the code compiles, maybe even runs once, but the design is broken. The fix is the same as for other hallucinations: don’t trust the output without verification. For code, verification means tests, review, and a clear model of what “correct” means.
Expect more tooling that checks generated code against specs or structural rules, and more patterns for “model proposes, system verifies.”
nJoy 😉