The obvious fix is to let the model guess and check its work.
The model reads zoloff and suggests a drug. Before you show that drug to anyone, you look it up in the database. If it is real, you trust it. If it is not, you throw it out. A guardrail on the output. Everyone reaches for this first. I did too.
It did not work. Here is why.
The guardrail that waves the wrong drug through
When the model read zoloff and answered Ambien, Ambien passed the check.
Ambien is a real drug. It is in the database. The lookup succeeded. The guardrail let it through, because the only question the guardrail knew how to ask was “is this a real drug,” and the answer was yes. It was the wrong real drug. A validator that checks whether something exists cannot catch a confident answer that happens to exist. And in a database of drugs, most wrong answers exist. That is what a drug database is.
So checking the model’s work after the fact buys you nothing here. The bad output is shaped exactly like a good one. The only test you have cannot tell them apart.
Two jobs wearing one coat
I had to stop thinking of this as a single task. And I had to keep one person in mind the whole time.
The doctor in the basement with no signal, holding my app like gospel because it is the only thing down there that answers. She will do exactly what it says. When it says the wrong thing, her lawyer already has my email address. So the real goal was never accuracy in the abstract. It was making sure that one woman never gets handed the wrong drug, so that one man never gets to hit send.
Turning zoloff into an FDA label is two jobs stacked on top of each other, and they are not the same difficulty. The old design handed the model both. That was the mistake.
The first job is to look at whatever a human typed and find the part that is trying to be a drug name. In “hey can you pull up zoloff for me,” the drug-shaped token is zoloff. The model is genuinely good at this. It is a language problem, and finding the meaningful span in a sentence is the thing a language model is actually for.
The second job is to decide which real drug zoloff means. This is where it reaches into the wrong neighborhood and comes back holding Ambien. This is the job it cannot be trusted with, because when it is wrong here it is wrong with total confidence, and the wrong answer is another drug that can hurt you.
The fix is to give it the first job and take away the second.
Keep the token, throw away the guess
So the model reads the sentence and hands back the literal token the user typed. zoloff. Not a correction. Not its best guess at the real name. The exact characters that were on the screen.
Then a deterministic matcher does the second job. It takes zoloff and compares it against every drug name in the database, letter by letter, and ranks them by how many single-character edits it takes to turn what you typed into each real name. Zoloft is one edit away. Ambien is not close to anything you typed. The matcher hands back the four nearest real names and the app shows them. The model never gets a vote on which drug it is.
Here is the part I had to sit with for a while. This fix does not make the model smarter. It is exactly as wrong as it was before.
I know that because I checked. I ran the real model on “please pull up metafarmin,” which is a mangled spelling of Metformin, a common diabetes drug. Given that input, the model produced the name warfarin. Warfarin is a blood thinner. It is the same species of confident wrong-neighborhood error that produced Ambien, and the model made it while I was sitting there watching.
But it also handed back the literal token metafarmin, because that is now the only thing I ask it for. The pipeline dropped warfarin on the floor and fed metafarmin to the matcher. Metformin is two edits from metafarmin. Warfarin is nowhere near it. The app showed Metformin. The word warfarin never left the function that thought of it.
That is the whole trick. I did not repair the model. I moved the line so that the thing the model is bad at now happens in a place the user can never see.
Why the matcher gets the dangerous job
The matcher is allowed to make the which-drug decision because it has a property the model does not have. It cannot invent.
It can only return names that are already in the database, and it returns them ranked by a number you can check by hand. There is no neighborhood for it to wander off into. Type something close to a real drug and you get that drug. Type something close to nothing and you get nothing, which is the honest answer and the safe one.
The model is probabilistic. Sooner or later it will confidently make something up, and you cannot predict which time. The matcher is deterministic. The same input gives the same output every time, and you can trace every step that led there. For the one decision in this app that can actually hurt someone, which drug this is, you want the boring machine that cannot lie over the clever one that sometimes does.
Where the line ends up
Probabilistic where it is safe, which is reading language. Deterministic where it is not, which is naming the drug.
The model reads. The matcher decides. Neither one is allowed to do the other’s job.
Next chapter, the matcher itself. It is not a single wall. It is a stack of them, and every layer in the stack is there because something got past the layer before it.
