The obvious fix is to let the model guess and check its work. The model reads zoloff and suggests a drug. Before you show that drug to anyone, you look it up in the database. If it is real, you trust it. If it is not, you throw it out. A guardrail on the output. Everyone reaches for this first. I did too. It did not work. Here is why. ...
Season 1 is over. Time to scale up. Six weeks ago, I started this blog with a simple question: what actually happens inside a language model? The answer turned into a six-post series where I built GPT-2 from scratch - 134 million parameters, 2.8 billion tokens, and a Colab session that crashed more often than it didn’t. I learned a lot. Not just about transformers and tokenizers, but about the thousand small decisions that determine whether your training run produces coherent English or expensive gibberish. I took training time from 90 minutes down to 21 minutes. I watched a random pile of floating-point numbers slowly learn that Paris is a city and “the” comes before nouns. ...
How I went from ‘cute toy model’ to ‘134 million parameters that need an A100 to breathe.’
