Hardware-aware dynamic speculative decoding

Speculative decoding (SD) is a technique used to accelerate large language model (LLM) inference without compromising quality. It utilizes a smaller draft model to propose multiple tokens, which are then verified by the larger target model in a single computational step. This method improves generation speed by exploiting the difference between a GPU's compute and memory bandwidth, especially when compute units are idle at small batch sizes. However, challenges arise in production environments with dynamic batch sizes, where SD can become slower than standard inference, and in reinforcement learning applications.