Troubleshooting GPU Scheduling Issues in Amazon EKS: A Practical Guide

Managing GPU-enabled infrastructure in Amazon Elastic Kubernetes Service (EKS) introduces operational complexities that often diverge from standard CPU and memory workload management. When deploying machine learning inferencing, data processing, or rendering pipelines on GPU nodes, infrastructure engineers frequently encounter hard scheduling limits. A common failure mode occurs when clusters reject new pod deployments, throwing cluster autoscaler errors such as pod didn’t trigger scale-up: 3 node(s) didn’t match Pod’s node affinity/selector, 6 max node group size reached. This technical guide outlines the diagnosis, root cause analysis, and step-by-step remediation procedures required to resolve GPU scheduling bottlenecks and implement resource sharing in production EKS environments.

Problem Statement & Impact

Production environments running containerized GPU workloads on AWS EC2 accelerated instances (such as g4dn.xlarge) frequently experience premature resource exhaustion. A single g4dn.xlarge instance provides 1 NVIDIA T4 Tensor Core GPU, 4 vCPUs, and 16 GiB of system memory. By default, the native Kubernetes and NVIDIA device plugin architectures treat a physical GPU as an indivisible, discrete resource. When a pod requests a GPU via the standard resource declaration (`nvidia.com/gpu: 1`), the Kubernetes scheduler maps the entire physical device to that single container.

Consequently, if an application requires only a fraction of the GPU’s compute capacity or VRAM—such as lightweight model inference or batch image processing—the remaining GPU capacity sits idle, yet inaccessible to other pods on the same node. Once the cluster runs out of physical GPUs, new pods remain in a Pending state. If configured with a cluster autoscaler, the system attempts to provision new EC2 instances until hitting maximum node group size limits or AWS account vCPU quotas, resulting in failed deployments, inflated cloud expenditures, and application downtime.

Root Cause Analysis & Architecture

The core mechanism driving this failure is the default behavior of the NVIDIA Device Plugin for Kubernetes. In a standard out-of-the-box configuration, the plugin exposes physical GPUs as integer quantities (e.g., nvidia.com/gpu: "1"). The kubelet tracks these resources as binary availability flags. A node with one GPU can accept exactly one pod requesting nvidia.com/gpu: 1. Any subsequent pod specifying that resource request will fail to schedule on that node, regardless of available CUDA cores or unused frame buffer memory.

To overcome this architectural limitation without purchasing additional hardware, engineers must implement GPU time-slicing. Time-slicing (or time-division multiplexing) allows the NVIDIA device plugin to expose a configured number of “virtual” or replica GPUs to the Kubernetes scheduler, mapping multiple container execution contexts to a single underlying physical GPU.

Architectural Considerations of Time-Slicing

• Lack of Hard Memory Isolation: Unlike NVIDIA Multi-Instance GPU (MIG) technology—which is restricted to A100, H100, and newer enterprise architectures—time-slicing does not enforce strict hardware-level memory partitioning. If a single sharing pod attempts to consume more than its fractional share of VRAM or crashes due to an out-of-memory (OOM) error, it can destabilize or impact other workloads sharing the same physical GPU.
• Scheduler Abstraction: The Kubernetes scheduler remains agnostic to the underlying hardware sharing; it simply evaluates the inflated replica count declared in the device plugin’s ConfigMap.
• Workload Suitability: Ideal for bursty, low-latency, or low-VRAM inference tasks where peak utilization rarely hits 100% simultaneously across all concurrent pods.

Step-by-Step Implementation & Remediation Guide

Resolving the scheduling bottleneck requires purging the default device plugin configuration, deploying a customized time-slicing ConfigMap, and applying correct node labeling and taints to ensure reliable pod placement.

Step 1: Uninstall the Existing NVIDIA Device Plugin

If the standard NVIDIA device plugin is already installed via Helm, remove the release to clear existing daemonsets and device registrations:

helm uninstall nvidia-device-plugin -n kube-system
kubectl delete daemonset nvidia-device-plugin-daemonset -n kube-system

Step 2: Define the GPU Time-Slicing ConfigMap

Create a Kubernetes ConfigMap that instructs the device plugin to multiply the available GPU capacity. The following manifest configures each physical GPU to support up to 4 replicas (replicas set to 4), effectively allowing 4 pods to share a single physical GPU on a g4dn.xlarge node.

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: kube-system
data:
  any-name.yaml: |
    version: v1
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 4

Apply this configuration to the cluster using kubectl apply -f time-slicing-config.yaml.

Step 3: Deploy the NVIDIA Device Plugin via Helm

Deploy the official NVIDIA device plugin chart, referencing the custom time-slicing ConfigMap created in the previous step. This deployment overrides the default resource discovery behavior.

helm repo add nvidia https://nvidia.github.io/k8s-device-plugin
helm repo update

helm install nvidia-device-plugin nvidia/nvidiadevice-plugin \
  --namespace kube-system \
  --set gfd.enabled=true \
  --set config.name=time-slicing-config \
  --set config.map=any-name.yaml

Step 4: Label and Taint GPU Nodes

To ensure that the scheduler correctly routes workloads and that non-GPU workloads do not occupy valuable accelerated instances, apply the required node labels and taints.

First, identify your GPU node names:

kubectl get nodes -l node.kubernetes.io/instance-type=g4dn.xlarge

Apply the required NVIDIA compatibility label so that newer device plugin versions can discover and register the nodes correctly:

kubectl label node <node-name> nvidia.com/gpu.present=true --overwrite

Apply a taint to the GPU nodes to prevent standard, non-accelerated pods from scheduling onto expensive GPU hardware:

kubectl taint nodes <node-name> nvidia.com/gpu=true:NoSchedule

To schedule workloads on these tainted nodes, update your application pod specifications to include matching tolerations and node selectors:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpu-inference-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: gpu-inference
  template:
    metadata:
      labels:
        app: gpu-inference
    spec:
      tolerations:
      - key: "nvidia.com/gpu"
        operator: "Exists"
        effect: "NoSchedule"
      containers:
      - name: inference-container
        image: your-registry/inference-image:latest
        resources:
          limits:
            nvidia.com/gpu: "1"
          requests:
            nvidia.com/gpu: "1"

Production Hardening & Edge Cases

Operating shared GPU architectures in production demands rigorous attention to base operating system configurations, AMI selection, and monitoring overhead.

• AMI Selection: Ensure nodes utilize EKS-optimized Amazon Linux 2 or Amazon Linux 2023 AMIs that feature pre-compiled NVIDIA drivers matching the CUDA version required by your container runtimes. Avoid unmanaged custom builds unless version parity between the host kernel driver and container toolkit is strictly maintained.
• Container Runtime Integration: Verify that the NVIDIA Container Toolkit is correctly configured in your containerd configuration (`/etc/containerd/config.toml`) so that containers can successfully interface with the allocated device endpoints.
• Monitoring VRAM Consumption: Because time-slicing does not isolate VRAM at the hardware level, deploy Prometheus alongside the NVIDIA Data Center GPU Monitor (DCGM Exporter) to track metric gauges such as DCGM_FI_DEV_FB_USED and DCGM_FI_DEV_GPU_UTIL. Set up alerting rules to detect memory leaks before an out-of-memory condition crashes shared node neighbors.

Diagnostic Verification & Checklist

Use the following verification matrix to confirm that GPU time-slicing is operating correctly across your EKS cluster post-remediation.