Breaking the Host Memory Bottleneck: How Peer Direct Transformed Gaudi’s Cloud Performance

Intel's Gaudi AI accelerators faced a severe performance bottleneck when deployed on AWS, as the cloud network topology forced all data through host memory, degrading distributed training performance by up to 50%. An engineering effort led to a solution called "Peer Direct" that restored direct, RDMA-like communication between accelerators across different nodes. The solution leveraged technologies like libfabric, DMA-BUF, and a custom communication library wrapper to bypass the host CPU and memory. This fix resulted in performance gains of up to 2x, enabling the successful launch of Gaudi on AWS DL1 instances. The project underscores the critical importance of optimizing the underlying data path and network topology for scalable AI training.