Maximizing GPU Efficiency: Rethinking Resource Allocation in Kubernetes

Understanding the GPU Utilization Issue

Many organizations are wrestling with a troubling pattern: despite owning powerful GPUs, their usage often hovers in the low double digits. I noticed this firsthand when running an nvidia-smi loop across an inference fleet, where many GPUs remained at only 12% to 30% utilization. Each running pod had reserved a full GPU but spent considerable time idling, waiting for inputs. This setup led to inflated cloud bills as companies paid for GPUs at a full rate while only tapping a fraction of their processing power.

This discrepancy between scheduled resources and actual utilization represents a significant leak in cloud-native AI budgets. It arises not from misconfiguration or user error, but from fundamentally flawed assumptions about how Kubernetes should manage these expensive computational assets, which differ starkly from traditional, stateless applications.

Kubernetes' Limitations with GPUs

Kubernetes is designed to work with the assumption that workloads are fungible—easy to scale up and down quickly. However, GPU inference pods defy this mold. They are costly, tied to substantial state data due to model weights, and require lengthy hydration times to load necessary data onto VRAM before they can execute any tasks.

Currently, Kubernetes manages GPUs using a rather simplistic device-plugin approach: it treats GPUs as extended resources labeled nvidia.com/gpu where each pod simply requests an integer count of GPUs. This method lacks nuance—the scheduler is blind to essential factors such as memory requirements or the potential benefits of sharing resources. Consequently, a pod requesting a single GPU may occupy a full card even when it could function efficiently on much less.

Strategies for Enhanced GPU Utilization

To tackle GPU wastage, the first step is to abandon the notion of treating GPUs as indivisible units. Instead, various mechanisms can be employed to allow multiple workloads to share a single physical GPU:

• Time-slicing: This software-based approach enables multiple processes to share the GPU in quick intervals. It requires a ConfigMap to enable and works across various NVIDIA GPUs. However, it lacks memory and fault isolation, meaning one overly demanding process can significantly affect others.
• Multi-process service (MPS): MPS allows for concurrent execution on the SMs, facilitating improved throughput and lower latency compared to time-slicing, while still lacking isolation guarantees.
• Multi-instance GPU (MIG): This hardware-based solution provides greater safety by partitioning a GPU into multiple isolated instances, each with its own memory and compute domain. It's available only on newer models like the A100 or H100 and relies on predefined profiles, making it a strategic decision for capacity planning.

Using any of these approaches enhances scheduling by providing better granularity than a simple count. For example, a deployment can now request specific MIG slices instead of an entire GPU:

apiVersion: apps/v1  
kind: Deployment  
metadata:  
  name: chat-small  
spec:  
  replicas: 4  
  template:  
    spec:  
      containers:  
        - name: server  
          image: my-registry/inference:latest  
          resources:  
            limits:  
              nvidia.com/mig-1g.5gb: "1"  
              memory: 12Gi  
            requests:  
              nvidia.com/mig-1g.5gb: "1"

Ultimately, the long-term solution lies in Dynamic Resource Allocation (DRA), which debuted in Kubernetes v1.34. For the first time, this allows the scheduler to request specific GPU attributes rather than simply an integer count, enabling finer control over how resources are utilized. However, DRA remains a work in progress, and its full capabilities won't be realized until future iterations.

When to Scale Efficiently

Despite implementing better sharing strategies, challenges remain regarding when to scale GPU workloads. Kubernetes' Horizontal Pod Autoscaler (HPA) typically relies on CPU usage as an indicator, which can mislead when CPU utilization is low while the GPU is overwhelmed at 100%. This misalignment can lead to delayed responses for users experiencing a backlog of requests.

A more accurate metric for scaling is backlog using queue depth, which helps visualize user demand before requests pile up. GPU utilization metrics sourced from the DCGM exporter into Prometheus can serve as effective guardrails. KEDA is particularly useful here, as it can scale applications to zero, something the standard HPA cannot do. When considering thresholds, ignore any generic CPU metrics from tutorials, instead targeting a more realistic 70-80% GPU utilization in production settings.

Addressing Cold Start Issues

Embracing GPU sharing and the scale-to-zero philosophy introduces another layer of complexity—the cold start problem. Unlike a typical container restart, bringing an inference pod from zero often entails loading large model weights that can take considerable time, contributing to response delays. High latency on cold starts can frustrate end-users and offset the benefits of scaling down.

To mitigate this, avoid embedding model weights directly into container images. Instead, keep the runtime image lightweight and stream model weights from object storage or a local cache during startup. This way, only one instance of each weight needs to load per node, effectively reducing redundant data pulls across replicas. Consider also maintaining a baseline number of replicas for latency-sensitive applications, reserving the option to scale to zero for less critical tasks.

The Bottom Line

It's time for organizations to rethink their GPU management strategies within Kubernetes. Treating a GPU as a mere integer, misjudging the implications of weight-loading, and relying solely on CPU metrics results in significant inefficiencies. Adopt strategies to divide GPU resources, scale intelligently based on relevant metrics, and streamline model weight management. These adjustments could turn inefficient workloads into well-utilized resources, aligning expenditure with actual performance more closely.