Unlocking asynchronicity in continuous batching

Synchronous batching for LLM inference is inefficient because the CPU and GPU take turns, leading to significant idle time and wasted compute resources. To maximize GPU utilization, asynchronous batching disentangles CPU batch preparation from GPU computation, allowing both to run in parallel. This is achieved using CUDA streams, which are ordered queues of GPU operations that can run concurrently if they are in different streams. By carefully managing these streams and synchronizing with CUDA events, it's possible to prepare the next batch on the CPU while the current batch is being processed on the GPU, significantly reducing idle time and boosting performance.