A great fun model
doesn't need to be
a big one.
95 controlled experiments. A single RTX 4090. Five minutes of training each. We tuned a 34-million-parameter autocomplete model to a 14.5% gain over baseline, spending nothing on scale and everything on science.
Type a prompt. Watch it finish your thought.
The real 34M-parameter network loads right in your browser and generates every completion on your device, with no API and no round-trip. This is the actual ONNX export of run_084, BPE tokenizer and all.
The goal was never a flawless model. After just five minutes of training, the completions will be rough. The point is what you're watching happen at all: a tiny model, trained on a shoestring, learning the shape of language well enough to finish a sentence.
Give every idea the
exact same five minutes.
Most scaling studies reward whoever throws the most compute at the problem. We did the opposite. Every one of the 95 runs got an identical 5-minute wall-clock budget on the same GPU, so the only thing that could win was a better decision, not a bigger wallet.
This was never about shipping a perfect tiny model. Five minutes can't buy fluency. It was about a simpler question: how much can a small model actually learn when every choice has to earn its place? The answer turned out to be surprisingly a lot.
Learning rate moved the needle most.
Holding architecture fixed at 6L×320, every increase in learning rate bought a lower loss, right up to 2e-3, far past where conventional wisdom says to stop. With the Muon optimizer, aggression pays.
Scale helps, until it doesn't.
More parameters lowered loss up to 34M. Past that, each step cost so much compute that a 50M model couldn't warm up inside five minutes, and lost to a model a third its size.
Top 10 of 95 runs.
| # | Run | Config | Params | val_bpb | vs. base |
|---|---|---|---|---|---|
| 1 | run_084 | 6L×640, LR=1e-3 | 34M | 1.3339 | -14.6% |
| 2 | run_050 | 6L×320, LR=2e-3 | 10M | 1.3511 | -13.5% |
| 3 | run_069 | 4L×384, LR=1e-3, batch=256 | 10M | 1.3593 | -13% |
| 4 | run_086 | 6L×512, seq=384, LR=1e-3 | 22M | 1.3666 | -12.5% |
| 5 | run_089 | 6L×320, LR=1e-3, batch=256 | 10M | 1.3655 | -12.5% |
| 6 | run_028 | 4L×384, LR=1e-3 | 10M | 1.3753 | -11.9% |
| 7 | run_049 | 6L×320, LR=1.5e-3 | 10M | 1.3777 | -11.8% |
| 8 | run_093 | 6L×512, LR=1.2e-3, batch=256 | 22M | 1.3790 | -11.7% |
| 9 | run_048 | 6L×320, LR=1.2e-3 | 10M | 1.3904 | -10.9% |
| 10 | run_001 | 6L×320, LR=3e-4 (baseline) | 10M | 1.5613 | · |
Six findings that beat brute force.
Learning rate dominates everything
More than architecture, attention pattern, or context length combined. Every step up from 3e-4 to 2e-3 improved results. Under a 5-minute budget, an aggressive LR pushes the model further down the loss curve before the clock runs out.
Smaller batches buy more steps
Batch 256 beat 512 and 1024 across the board. With a fixed wall-clock, gradient velocity beats gradient precision, and more updates per minute wins.
Wide and shallow learns faster
At equal parameters, 4 layers × 384 dim outran 8 layers × 256 dim. Wider representations propagate signal in fewer steps than deeper stacks.
Scale pays off, to a point
34M params (6L×640) was the sweet spot. At 50M the per-step cost outweighed the added capacity. The crossover for a 5-minute RTX 4090 sits near 30–40M.
Long context is a tax you can't afford
seq=512 lost to seq=256 in every sweep. Quadratic attention cost means fewer examples seen. Under time pressure, shorter wins.
Sliding windows beat full attention
SSSL (3 sliding, 1 full) consistently beat all-full LLLL. Cheaper layers mean more steps per minute, with enough context for prompt-length text.
The full ablation.
| Change | Effect | Verdict |
|---|---|---|
| LR: 3e-4 → 1e-3 | −0.14 bpb (large) | ✓ helps |
| LR: 1e-3 → 2e-3 | −0.07 bpb (medium) | ✓ helps |
| Batch: 512 → 256 | −0.05 bpb (medium) | ✓ helps |
| Params: 10M → 34M | −0.02 bpb (small) | ✓ helps |
| Params: 34M → 50M | +0.17 bpb (large) | ✗ hurts |
| seq: 256 → 512 | +0.08 bpb (medium) | ✗ hurts |
| Dropout: 0 → 0.1 | +0.13 bpb (large) | ✗ hurts |
| Attention: SSSL → LLLL | +0.05–0.09 bpb | ✗ hurts |
autocomplete-tiny
A 34M-parameter causal transformer with sliding-window attention, trained with the Muon optimizer on 500,000 real prompts. Small enough to run in a browser tab, yet sharp enough to finish your sentence.