Writing

Training a 30M-param GPT from scratch

Multi-head self-attention, positional encoding, training on tinyStories with JAX / Optax / Orbax on a T4. What broke, what worked, what I'd do differently.