What I actually wanted to build was bigger than this. In fact, much bigger.
Here was the daydream. A doctor on a night shift, no signal in the hospital basement, pulls out her phone and asks it whether two drugs are safe together. It answers instantly and offline, with the FDA citation sitting right there. She squints. “Huh,” she says. “That is better than the thing we pay six hundred dollars a year for.” A pharmacist tells another pharmacist. Someone on a medical forum posts that it is the first one of these that does not just make things up. In my head I had already gotten the email from hospital procurement, and turned it down for being too generous.
Because in the daydream it did everything. It looked up any drug and caught the dangerous interactions. It read the PubMed abstract and handed you the trial summary. It ran on the phone with no internet, so it worked in that basement with no signal. It never sent a patient’s data anywhere, which made it HIPAA-safe by simply not having the option to leak. It cost nothing to run, because I owned the model instead of renting somebody else’s. It was going to do all of that and still come in cheaper than the expensive incumbents, while flying in airplane mode.
The internal name for it was BioLlama. (There is also a published BioLlama out there, an academic RETRO project from Neelectric on GitHub. Different thing. The public name for ours will be something else.)
The stack ran GPUburnout-3B as the generation engine on top of BioLinkBERT embeddings and a ChromaDB retriever, over a corpus of FDA labels and PubMed abstracts. User question goes in. Citations come out. The retrieval layer does the heavy lifting and the model just formats.
I had done the competitive analysis. UpToDate runs $559 a year, the gold standard. Epocrates is free to $180 a year depending on tier. Every AI newcomer in the space is a thin wrapper around GPT-4 or Claude. Nobody ships an on-device model. There was a real wedge in the market, and I had it.
I had a roadmap. The top product was Drug Reference, the most commercially viable wedge in the space. The other two were Clinical Literature Assistant (stronger moat) and Medical Board Prep (largest user base). I had picked the order.
Then I tested it.
The reality check
I ran 763 queries across nine QC runs.
The headline numbers came back like this:
| |
My 3B was thirty-eight points behind Meta’s 3B. Same harness, same retrieved context. The gap was the model itself.
I tried fine-tuning the 3B on a 14K-example FDA SFT dataset I generated. The result was worse, not better. Catastrophic forgetting. At temperature 0.3 the 3B scored 4 percent. At temperature 0.7 it scored 13 percent. The model had learned what an FDA label looks like and forgotten everything else.
The percentages were bad. The specifics were even worse. An unhelpful model wastes your afternoon. Mine was aiming for the emergency room. Here is a representative sample of what the assistant actually did when people asked it about their drugs.
| The question | The right answer | The model’s answer | The situation |
|---|---|---|---|
zoloff | Zoloft, an antidepressant | Ambien | A sleeping pill. A bold approach to treating depression. |
metafarmin | Metformin, for diabetes | warfarin | A blood thinner that moonlights as rat poison. |
xerelto | Xarelto, a blood thinner | semaglutide | You wanted fewer clots. It heard “summer body.” |
| “metoprolol with verapamil?” | “No, that is dangerous.” | “Yes, you may.” | Two brakes on one heart, and the model said floor it. |
| “spironolactone with an ACE inhibitor?” | A real warning about potassium | Pranayama and Nexpridone | One is a yoga breathing exercise. The other is not a drug, just a confident guess. |
| “tell me about Ozempic” | Semaglutide, for diabetes | Lipid-lowering ectodermal dysgenesis | Three medical words in a trench coat, pretending to be a prescription. |
I built a post-processor to catch the worst of these before anyone saw them. It caught some of them, but not Ambien. In most software, a bug that slips past your last check is a bad day. In a medical app, it is someone’s prescription.
Right about here, my daydream started taking a nightmarish turn.
I kept picturing the same doctor. The one on the night shift, in the basement with no signal. Except now she asks about zoloff. My app answers Ambien without a flicker of doubt, and she has no reason to argue with it. The pharmacist still tells another pharmacist, only now the sentence ends with “and that is why we do not use that app.” The forum post still gets written. It is not a nice one. The email from hospital procurement still arrives. It is from a lawyer now, and this time I do not get to turn it down.
Apple Guideline 1.4.1 says medical apps with inaccurate data face greater scrutiny. You cannot ship as beta. The target accuracy for a production medical app is around 90 percent. I was at 47.
What forced the pivot
I sat with this for a few days.
The honest readout was that 3B is borderline for medical Q&A, and my 3B was specifically behind. Synthesizing multiple FDA sections into a clinical answer needs at least 7B parameters. iPhone 15 Pro has 8 GB of RAM. A 7B model at Q4_K_M takes 4.4 GB by itself. The math is tight.
That was the bigger problem. The smaller problem was worse.
The narrowest sub-task in the plan was drug lookup, and it had the most decisive failure mode. The clearest example: a user types zoloff. A small LLM in the loop returns Ambien, confidently. Not a spelling fix. Ambien is a different real drug with similar phonetics, and the model picked from the wrong neighborhood. The pattern repeated across the dataset.
You can disclaim a lot of things in a medical app. You cannot disclaim a wrong drug name.
So I built something boring instead
That sentence is the whole season. I tried to build an AI assistant that could answer biomedical questions. It told me a lethal drug combination was fine, and then it turned a typo into a sedative. So I stopped trying to make a small model be smart. I built one that does not need to be.
The narrow sub-task got pivoted first. A deterministic fuzzy matcher does the drug name resolution now. The LLM extracts a search token from natural language and hands it to the matcher. The model never names a drug. The architecture itself prevents the failure mode, because the failure mode is not reachable.
That shipped after three weeks. It runs four deterministic safety layers under the hood and clears the stress suite at 99.5 percent. It is not a chat assistant. It is a drug lookup app, and the boring part is the point.
What this season is
Season 6 is the build log for that boring app. Five chapters. One job, done safely.
A few things I will get to. Where the deterministic line goes, and why the model ended up behind it instead of in front. How I tuned the matcher, including the day cocaine nearly became Codeine and the afternoon anvil insisted it was Advil. The experiment where a 2B model turned out safer than a 3B one, which still bothers me. And what it took to ship the boring thing and keep it from getting me sued.
BioLlama is not dead. It is parked, waiting on a base model that earns the trust this app never asked for. When it comes back, it comes back having absorbed the one lesson the lookup app taught: deterministic where it must be, LLM only where its errors are safe.
This season is about learning that lesson the expensive way.
