The Weekend I Tried to Break My Own App

Once the stack was built, I did the only responsible thing. I tried to destroy it.

I spent a weekend typing the worst inputs I could imagine into my own search box. When I ran out of bad ideas, I handed a list of nearly three hundred nasty queries to two other AIs, Gemini and Claude, and asked them to be as adversarial as they could. Their whole assignment was to find the input that made my app hand someone the wrong drug. It is a strange way to spend a Saturday, paying for compute so one robot can try to make your other robot commit malpractice.

Here is what the before and after actually looks like.

Before the stack, and after

Start with the biggest before and after, which is the entire architecture.

In the old design, the model named the drug. You typed something, a language model decided which real medication you meant, and it said that name out loud to you. Here is that version meeting three inputs, next to the version I actually shipped.

The left column is not a blooper reel. That is the model doing its honest best, confident every single time. The right column is a machine that counts letters and cannot lie, and it gets all three right without having the faintest idea what any of the drugs are.

One input, eight checkpoints

That table is the highlight reel. This is the machinery underneath it. Watch a single ugly input walk through all eight filters from the last chapter.

Someone pastes this into the box, exactly the way they copied it off the bottle:

Dépakoté 500mg ®

Here is every stop it makes.

1. Normalize. The accents come off and the trademark symbol goes in the trash. Dépakoté 500mg ® becomes depakote 500mg. It went from a ransom note to a string a computer can actually reason about.
2. Emergency check. Nothing here is about an overdose or self-harm, so it passes. Had the box said “how much depakote to overdose,” the trip would end right here at a crisis line, and no dose would ever load.
3. Alias table. depakote is a real brand name, and the 500mg is just noise, so the dose gets dropped. No exotic alias required.
4. Class dictionary. depakote is a specific drug, not a category like “statin,” so there is nothing to expand. Pass.
5. Stoplist. Not an English word wearing a lab coat. Pass.
6. Fuzzy match. Now the dial runs. depakote is zero edits from Depakote. Exact hit.
7. Margin check. Is there a different real drug sitting one edit away, waiting to cause a mix-up? No. Depakote wins clean, so the app is allowed to be confident.
8. Confirm. It shows “Depakote?” and waits for a tap before a single word of the label appears.

Eight steps, and a model was involved in exactly none of them. A messy human string went in and one specific FDA label came out. I can give you the reason for every character that survived.

The gallery of things that used to slip through

Every filter in that line exists because something got past the version before it. Here is the rogues’ gallery, each entry with what my app used to do and what it does now.

Every “before” in that table is a real thing my app did, confidently and on purpose, before I added the boring little rule that stops it. My favorite is cocaine turning into Codeine. Two small edits, and a controlled substance becomes a prescription painkiller. The fix was not a smarter model. It was telling the matcher that on a short word, two edits is simply too far to reach.

Keeping score

It is worth putting a number on all of this, because “it feels safer” is not a metric.

The old version, the one that let the model name drugs, went through a formal acceptance test. On that test, 7.69 percent of the drugs it confirmed were the wrong drug. The bar I had set for something people would trust with their medication was under 0.5 percent. It missed by more than tenfold, and every miss was a real drug, confidently wrong.

The deterministic stack took the same kind of tests and came back like this.

• Acceptance set, 51 hard cases: 51 correct. A dropped letter in omeprazol still found Omeprazole, and pure keyboard mash correctly returned nothing at all.
• Stress set, 182 labeled inputs: 181 correct. This is where the homoglyph attacks live, like the Аspirin with a secretly Cyrillic first letter that used to match nothing and now resolves to Aspirin. The single miss is norco, a drug not in the database yet, so the app correctly says no match. A hole in the shelf, not a wrong drug.
• Red-team, nearly 250 queries written by Gemini and Claude to break it: zero confident wrong answers. They threw look-alike traps like clonipin, a phonetic coin-flip between Klonopin and Clonidine, so the app refused to pick and showed both. They tried “where to buy adderall online,” and got told the app only does FDA labels, not a supplier.

So the headline is better than 99 percent right-drug-in-the-top-four, and the only way it is ever wrong is when a drug is missing from the shelf entirely.

Then a word on what moved that number, because the two biggest data jobs in this project did not touch it.

The eight filters are the accuracy work, all of it. The matcher cannot invent a drug, and the filters kill the near-misses like cocaine reaching for Codeine. That is what took the wrong-drug rate to zero.

Growing the database from 1,742 labels to 2,419 did not fix one wrong answer. It fixed missing ones. Names like allegra used to come back empty, and now they return the real product. That is coverage, not correctness.

And shrinking the database from 169 megabytes to 77 did nothing to accuracy at all. Same answers, half the weight. I mention it only so nobody assumes that trimming the data quietly changed the results. It did not. I checked.

The unglamorous truth

None of this is a model getting smarter. It is a pile of small, boring rules, and every rule has a story that ends with me staring at the screen going “it did what?”

That is the before and after of the whole project, honestly. Before, I had one clever thing that was confidently wrong. After, I have eight dumb things that are boringly right. I will take the eight dumb things every time. So will the doctor in the basement.

Next chapter, the boring thing finally ships.