6 Things I Learned Building LLMs From Scratch That No Tutorial Teaches You

Building large language models from scratch reveals several non-obvious architectural choices crucial for optimization and performance. Rank-stabilized LoRA (RsLoRA) is shown to be superior to standard LoRA for fine-tuning, as it maintains stable weight updates when scaling the rank. Rotary Positional Embeddings (RoPE) are preferred over older methods because they encode position by rotating query and key matrices without altering the token embeddings or adding parameters. Other key insights include the trade-offs between Pre-LayerNorm for stability and Post-LayerNorm for performance, the diminishing returns of weight tying in very large models, and the essential role of KV-caching for efficient inference.