Optimizing Data Transfer in AI/ML Workloads

AI/ML workloads can suffer from performance bottlenecks when the GPU idles while waiting for data from the CPU, an issue known as GPU starvation. This post focuses on data transfer bottlenecks and demonstrates how to identify and resolve them using the NVIDIA Nsight™ Systems (nsys) profiler. While the PyTorch Profiler is useful for framework-level analysis, nsys offers a more detailed system-level view of hardware and OS activities, making it better for diagnosing complex issues. A toy PyTorch model with a synthetic dataset is used to create an intentional bottleneck, and the article details the setup and code for profiling the workload with nsys to analyze system performance.