One token to corrupt them all: a vLLM debugging tale

A Jamba series LLM was found to occasionally generate high-confidence gibberish during reinforcement learning training, a problem isolated to the vLLM inference framework. The bug was elusive, occurring only once in about a thousand prompts, and did not appear when using standard Hugging Face transformers. A systematic debugging process was established by comparing logprobs from vLLM's output against a reference generated by transformers to reliably detect the state corruption. The investigation led deep into vLLM's internals, revealing a critical issue in how its scheduler and cache management handled the Mamba model architecture. Ultimately, the root cause was identified and a fix was contributed back to the vLLM project.