Why Your Amazon EKS Pods Are Stuck in Pending Despite “Available” IPs

When running scaled Kubernetes workloads on Amazon Elastic Kubernetes Service (Amazon EKS), infrastructure operators frequently encounter IP address exhaustion within their Virtual Private Cloud (VPC). To mitigate this limitation and support higher pod density per worker node, AWS introduced prefix delegation within the Amazon VPC Container Network Interface (CNI) plugin. Instead of binding individual secondary private IPv4 addresses to Elastic Network Interfaces (ENIs), prefix delegation assigns contiguous blocks of addresses—specifically, /28 CIDR blocks containing 16 IP addresses each—to node ENIs. This architecture dramatically increases maximum pod capacities per node instance type.

However, this optimization introduces a severe architectural failure mode: subnet fragmentation. Operations teams frequently observe pods stuck in the Pending state with the error message FailedCreatePodSandBox or failed to assign IP address, despite the AWS Management Console indicating hundreds of “available” IP addresses in the subnet. The management console displays the raw count of unassigned individual IP addresses, not the availability of contiguous /28 blocks. When a worker node requests a new /28 prefix block and none exist contiguously due to scattered allocations across legacy pods, scale-outs halt completely.

Root Cause Analysis & VPC CNI Mechanics

To understand why pods remain stuck in Pending despite high free IP metrics, engineers must analyze how the Amazon VPC CNI plugin interacts with the AWS VPC API and the Linux kernel network stack. In standard mode without prefix delegation, each Elastic Network Interface attached to an EC2 instance consumes individual secondary private IPv4 addresses. Every instance type has a hard ceiling on the maximum number of ENIs and secondary IPs per ENI it can support.

Enabling prefix delegation alters this resource model. The Amazon VPC CNI (running as the aws-node DaemonSet) requests a /28 prefix instead of single IPs. A /28 prefix contains 16 IP addresses, where the first IP is typically reserved as the network address, the last as the broadcast address, and the remaining 14 are available for pod allocation. The underlying AWS VPC API must find a contiguous block of 16 addresses within the subnet’s CIDR range that are completely unallocated.

Consider a subnet provisioned with a /25 netmask, which yields a total of 128 IP addresses. Divided mathematically, a /25 subnet can accommodate exactly eight distinct /28 blocks:

• Block 0: 10.100.0.0/28
• Block 1: 10.100.0.16/28
• Block 2: 10.100.0.32/28
• Block 3: 10.100.0.48/28
• Block 4: 10.100.0.64/28
• Block 5: 10.100.0.80/28
• Block 6: 10.100.0.96/28
• Block 7: 10.100.0.112/28

If dynamic pod scaling and termination cycles cause long-lived pods or persistent workloads to scatter across these blocks—for instance, one IP occupied in Block 0, two in Block 1, and so forth—the AWS VPC IPAM (IP Address Management) engine cannot assemble a clean /28 block. Even if 40 total individual IP addresses are unallocated across the subnet, if those 40 IPs are fragmented across every available /28 boundary, the system possesses zero assignable prefix blocks. The aws-node DaemonSet fails to attach a new prefix to the ENI, and the Kubernetes scheduler leaves incoming pods in the Pending queue indefinitely.

Step-by-Step Implementation & Diagnostics

Resolving this condition requires executing systematic diagnostic procedures using the AWS CLI and kubectl to inspect actual prefix allocations rather than relying on high-level console metrics.

Step 1: Validate CNI Configuration and Prefix Delegation Status

First, verify that prefix delegation is actively enabled within your cluster’s amazon-vpc-cni configuration. Execute the following command to check the configuration map:

kubectl describe daemonset aws-node -n kube-system | grep -i ENABLE_PREFIX_DELEGATION

If the output returns true, the node daemon is configured to request /28 blocks. Next, inspect the current environment variables in the running aws-node pods to confirm:

kubectl get daemonset aws-node -n kube-system -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="ENABLE_PREFIX_DELEGATION")]}'

Step 2: Isolate Pending Pod Error States

Identify the exact failure reason for the pods stuck in initialization by querying the Kubernetes API server for events associated with pending workloads:

kubectl get pods --all-namespaces | grep Pending
kubectl describe pod  -n 

Look specifically for the warning events generated by the kubelet and vpc-resource-controller:

Type     Reason            Age   From               Message
----     ------            ----  ----               -------
Warning  FailedCreatePodSandBox  2m    kubelet            Failed to create sandbox: rpc error: code = Unknown desc = Failed to setup network: add sandbox to network failed: plugin type="aws-cni" failed (add): add to network failed: failed to assign IP address

Step 3: Audit Subnet Fragmentation via AWS CLI

To prove that subnet fragmentation is the root cause, query the AWS EC2 API to retrieve all network interfaces and their currently assigned secondary IPs and prefixes within the affected subnet. Execute the following bash script to map allocated IPs against /28 boundaries:

#!/bin/bash
SUBNET_ID="subnet-0123456789abcdef0"
aws ec2 describe-network-interfaces \
    --filters Name=subnet-id,Values=$SUBNET_ID \
    --query 'NetworkInterfaces[*].[NetworkInterfaceId, PrivateIpAddress, PrivateIpAddresses[*].PrivateIpAddress, PrivateIpAddresses[*].Primary]' \
    --output json

To inspect assigned prefixes directly on the ENIs, run:

aws ec2 describe-network-interfaces \
    --filters Name=subnet-id,Values=$SUBNET_ID \
    --query 'NetworkInterfaces[*].Ipv4Prefixes[*].Ipv4Prefix' \
    --output json

If the returned list of active prefixes is significantly lower than the node capacity requirements, yet individual IP addresses show high usage in the VPC console, the subnet is severely fragmented.

Production Hardening & Architectural Remediation

Addressing subnet fragmentation requires immediate tactical remediation combined with long-term architectural redesign of your VPC infrastructure.

Immediate Remediation: Draining and Redeploying Nodes

When a subnet is completely fragmented, forcing the release of allocated blocks is difficult because active Kubernetes pods hold leases on the scattered IP addresses. To resolve this without manual IP surgery:

1. Provision a New Subnet: Create a new subnet (recommended minimum size of /24) within the same Availability Zone and add it to your EKS cluster’s node group subnet association list.
2. Cordon and Drain Existing Nodes: Gracefully migrate workloads away from the fragmented nodes to force the release of ENIs and their associated IP blocks.

kubectl cordon 
kubectl drain  --ignore-daemonsets --delete-emptydir-data --force

When the nodes are terminated, AWS releases the ENIs and their corresponding /28 prefix blocks back to the VPC IPAM pool, coalescing the address space.

Long-Term Subnet Sizing Strategy

Never run production Amazon EKS clusters with prefix delegation enabled in subnets smaller than a /24 (256 IP addresses). A /24 subnet provides 16 distinct /28 blocks, vastly reducing the probability of unrecoverable fragmentation during high-churn autoscaling events. For large-scale enterprise clusters supporting thousands of pods, provision /22 or /23 subnets.

Additionally, adjust the WARM_PREFIX_TARGET and WARM_IP_TARGET environment variables in the aws-node DaemonSet to control how aggressively the CNI pre-allocates resources. Setting a controlled warm prefix target prevents excessive over-allocation:

kubectl set env daemonset aws-node -n kube-system WARM_PREFIX_TARGET=1

Diagnostic Verification & Monitoring Matrix

To ensure cluster stability and preemptively detect IP exhaustion and fragmentation before workloads fail, implement continuous monitoring across the infrastructure metrics outlined below.

Regularly audit your Kubernetes cluster network metrics and VPC IP address spaces to ensure continuous operational resilience, preventing unexpected deployment blockages during high-traffic scaling events.