Making Knowledge Distillation Cheap Enough to Run at Scale

Knowledge distillation is a common but expensive method for compressing large teacher models into smaller student models, primarily due to massive VRAM requirements. A new approach makes this process more efficient with two key changes: offline distillation and a fused, chunked KL loss. Offline distillation involves caching the teacher's top-K logits once, removing the need to keep the large teacher model in memory during the student's training. The fused, chunked KL loss is a memory-efficient algorithm that avoids materializing the enormous vocabulary-by-sequence-length matrix, drastically cutting VRAM usage and making large-scale distillation practical on less hardware.