One token to corrupt them all: a vLLM debugging tale

While developing a new Jamba model, a rare bug was discovered that caused it to generate gibberish approximately once per thousand prompts. The process of finding and fixing this issue required a deep investigation into the vLLM codebase. This debugging effort uncovered how vLLM's scheduler interacts with various model architectures. The team shares their specific fix and the broader lessons learned to assist others who may face similar challenges when debugging large-scale AI systems.