Reducing LLM training waste with model-agnostic padding minimization

Padding is a significant source of wasted compute when training large language models, particularly in online-RL training. While sequence packing is a common solution for transformer models, it is not easily applicable to hybrid architectures like Transformer-SSM models. A model-agnostic approach using micro-batch-level truncation and padding-aware micro-batching can mitigate this issue. This method successfully eliminates approximately 90% of padding-related overhead, providing a simple and broadly applicable way to improve training efficiency across various model architectures.