A high-level guide to GPU utilization: allocation, kernel, and Model FLOP/s utilization

Modal published a guide distinguishing three things commonly called "GPU utilization": GPU Allocation Utilization (GPU-seconds running application code over GPU-seconds paid for), GPU Kernel Utilization (fraction of time running code on GPUs), and Model FLOP/s Utilization (share of theoretical arithmetic bandwidth used). The guide focuses on neural network inference workloads, which Modal calls a revenue center rather than a cost center.