The deterministic matcher was supposed to be the safe part. It mostly is. But safe does not mean finished, and the matcher had failure modes of its own. Most of the work in this project was not building it. It was tuning it. The dial The matcher has one main dial, but the dial only makes sense once you have watched the thing work. It is worth seeing once, because the whole safety argument rests on how dumb and how legible this machine is. ...
Time to face the music Training a language model is the fun part. You watch the loss drop, you generate text samples that are slightly less incoherent than yesterday’s, you tell yourself “look, it almost knows what France is.” It’s addictive. It’s rewarding. It also tells you absolutely nothing about how good your model actually is. Benchmarking is where the universe hands you a report card you didn’t ask for. ...
Final loss curves, the damage to my compute budget, and 22 lessons I paid dearly to learn.
