The Infrastructure Behind Making Local LLM Agents Actually Useful

Running a local LLM agent for scientific workflows presents challenges with speed and context length, as an agent built for single-cell RNA-seq analysis initially took 10-15 seconds per iteration and would crash from context overflow. To solve these issues, several infrastructure optimizations were implemented using the vLLM inference server. Inference speed was significantly improved by using CUDA graphs to reduce kernel dispatch overhead and FP8 quantization to decrease the model's memory footprint, thereby allowing for a larger KV cache. These changes, combined with tensor parallelism, enable long analysis sessions to run efficiently on local hardware like A100 and H100 GPUs without context overflow errors.