AI in Multiple GPUs: Understanding the Host and Device Paradigm

The host-device paradigm governs how CPUs and GPUs interact in AI, with the CPU (host) managing logic and the GPU (device) executing parallel computations. Commands are sent asynchronously from the host to the device via CUDA Streams, which are ordered queues of operations that enable the CPU to work on other tasks while the GPU is busy. Using multiple streams allows for concurrent execution, such as overlapping data transfers with model computations, but requires careful synchronization to manage dependencies. A key performance bottleneck is host-device synchronization, where the CPU is forced to wait for a result from the GPU, so efficient code minimizes these blocking events. Scaling to multiple GPUs involves assigning each process a unique 'rank' that controls a single device, setting the stage for distributed computing.