KEDA Autoscaling on Amazon EKS using SQS + EKS Pod Identity: A Practical Guide

Problem Statement & Impact

Cloud-native architectures running event-driven workloads frequently encounter efficiency bottlenecks when relying solely on native CPU or memory metrics within Kubernetes. In environments processing asynchronous payloads via message queues, infrastructure scaling must directly correlate with queue depth rather than compute resource saturation. When compute scaling lags behind incoming message rates, latency spikes degrade performance, leading to dropped tasks, processing timeouts, and failed downstream dependencies.

In traditional implementations, bridging external event sources like Amazon SQS to Kubernetes Horizontal Pod Autoscalers (HPA) required storing long-lived static AWS IAM credentials within Kubernetes Secrets or injecting environment variables directly into pod definitions. This pattern introduces severe security vulnerabilities. Compromised cluster nodes or arbitrary code execution vulnerabilities in application containers expose static access keys, allowing attackers to escalate privileges and access unauthorized AWS resources across the entire account.

To eliminate these risks, modern Amazon EKS deployments must utilize Amazon EKS Pod Identity. This mechanism maps AWS IAM roles directly to Kubernetes service accounts using cryptographically secure tokens managed by the EKS Pod Identity agent. Implementing this architecture ensures that worker pods and the autoscaling control plane assume least-privilege IAM roles dynamically, completely removing static credentials from the cluster state.

Root Cause Analysis & Architecture

Standard Kubernetes HPAs scale workloads based on metrics exposed via the Metrics Server, which only tracks resource utilization such as CPU and memory consumption. For stateless web frontends, this approach is sufficient. However, for worker pods processing asynchronous queues, CPU utilization often remains low or fluctuates wildly while thousands of messages accumulate in an upstream queue.

KEDA (Kubernetes-based Event Driven Autoscaler) resolves this limitation by acting as both a custom metrics provider and a controller. The KEDA operator queries external event sources directly—such as an Amazon SQS queue’s ApproximateNumberOfMessages attribute—and registers these values as custom metrics in the Kubernetes API server via the metrics.k8s.io or custom.metrics.k8s.io APIs. The native HPA controller then consumes these metrics to adjust the replica count of the target Deployment.

The authentication workflow relies on the AWS SDK within the KEDA operator and worker pods communicating with the EKS Pod Identity webhook and local agent. When a pod starts, the EKS Pod Identity mutating webhook injects environment variables (`AWS_CONTAINER_CREDENTIALS_FULL_URI` and `AWS_ROLE_SESSION_NAME`) and mounts a projected volume containing an authorization token. The application or KEDA scaler uses this token to request temporary session credentials from the local EKS Pod Identity agent daemonset, which calls `AssumeRole` via the AWS STS API on behalf of the IAM role associated with the Kubernetes service account.

Step-by-Step Implementation & Remediation Guide

This implementation guide configures an Amazon SQS queue, installs the EKS Pod Identity agent, deploys KEDA, establishes scoped IAM trust relationships, and deploys an autoscaling SQS worker application.

Step 1 — Create the SQS Queue

Execute the following AWS CLI commands to create a standard SQS queue and capture its configuration parameters:

export AWS_REGION="us-east-1"
export QUEUE_NAME="keda-work-queue"

# Create the SQS queue
aws sqs create-queue --queue-name ${QUEUE_NAME} --region ${AWS_REGION}

# Capture the Queue URL
export SQS_QUEUE_URL=$(aws sqs get-queue-url --queue-name ${QUEUE_NAME} --region ${AWS_REGION} --output text)
echo "Queue URL: ${SQS_QUEUE_URL}"

Step 2 — Verify or Install the EKS Pod Identity Agent

Check if the EKS Pod Identity agent add-on is active in your cluster. If using EKS Auto Mode, this is pre-installed.

export CLUSTER_NAME="your-eks-cluster-name"

aws eks describe-addon --cluster-name ${CLUSTER_NAME} --addon-name eks-pod-identity-agent --region ${AWS_REGION}

If the add-on is not present, install it via the AWS CLI:

aws eks create-addon --cluster-name ${CLUSTER_NAME} --addon-name eks-pod-identity-agent --region ${AWS_REGION}

Step 3 — Install KEDA via Helm

Add the KEDA Helm repository, update local charts, and deploy KEDA into a dedicated namespace:

helm repo add kedacore https://kedacore.github.io/charts
helm repo update

kubectl create namespace keda
helm install keda kedacore/keda --namespace keda --set metricsServer.useServiceAccount=true

Step 4 — Create IAM Roles and Trust Policies for Pod Identity

Create a trust policy file (`trust-policy.json`) that permits the EKS Pod Identity principal to assume the role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "pods.eks.amazonaws.com"
      },
      "Action": [
        "sts:AssumeRole",
        "sts:TagSession"
      ]
    }
  ]
}

Create the IAM role for the KEDA operator and attach permissions to read SQS queue metrics:

aws iam create-role --role-name KedaOperatorSQSRole --assume-role-policy-document file://trust-policy.json

# Attach policy allowing queue inspection
aws iam put-role-policy --role-name KedaOperatorSQSRole --policy-name SQSReadMetricsPolicy --policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:GetQueueAttributes",
        "sqs:GetQueueUrl",
        "sqs:ListQueues"
      ],
      "Resource": "*"
    }
  ]
}'

Create a separate IAM role for the worker application to consume and delete messages:

aws iam create-role --role-name SqsWorkerAppRole --assume-role-policy-document file://trust-policy.json

aws iam put-role-policy --role-name SqsWorkerAppRole --policy-name SQSWorkerPermissionsPolicy --policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes",
        "sqs:GetQueueUrl"
      ],
      "Resource": "*"
    }
  ]
}'

Step 5 — Configure Kubernetes Service Accounts and Pod Identity Associations

Create the application namespace and service account:

kubectl create namespace app-workload
kubectl create serviceaccount sqs-worker-sa --namespace app-workload
kubectl create serviceaccount keda-operator-sa --namespace keda

Associate the IAM roles with the respective Kubernetes service accounts using the EKS Pod Identity API:

export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

aws eks create-pod-identity-association \
  --cluster-name ${CLUSTER_NAME} \
  --namespace app-workload \
  --service-account sqs-worker-sa \
  --role-arn arn:aws:iam::${ACCOUNT_ID}:role/SqsWorkerAppRole \
  --region ${AWS_REGION}

aws eks create-pod-identity-association \
  --cluster-name ${CLUSTER_NAME} \
  --namespace keda \
  --service-account keda-operator \
  --role-arn arn:aws:iam::${ACCOUNT_ID}:role/KedaOperatorSQSRole \
  --region ${AWS_REGION}

Step 6 — Deploy the SQS Worker Application

Create the manifest for the worker deployment (`worker-deployment.yaml`). Replace `` with your actual queue URL.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sqs-worker
  namespace: app-workload
spec:
  replicas: 0
  selector:
    matchLabels:
      app: sqs-worker
  template:
    metadata:
      labels:
        app: sqs-worker
    spec:
      serviceAccountName: sqs-worker-sa
      containers:
      - name: worker
        image: amazon/aws-cli:2.15.0
        command: ["/bin/sh", "-c"]
        args:
          - |
            while true; do
              MESSAGE=$(aws sqs receive-message --queue-url  --max-number-of-messages 1 --wait-time-seconds 5 --region us-east-1 --output json)
              RECEIVE_HANDLE=$(echo $MESSAGE | grep -o '"ReceiptHandle": "[^"]*' | grep -o '[^"]*$')
              if [ ! -z "$RECEIVE_HANDLE" ]; then
                echo "Processing message..."
                aws sqs delete-message --queue-url  --receipt-handle "$RECEIVE_HANDLE" --region us-east-1
              fi
            done
        resources:
          limits:
            cpu: 500m
            memory: 512Mi
          requests:
            cpu: 100m
            memory: 128Mi

Apply the worker deployment configuration:

kubectl apply -f worker-deployment.yaml

Step 7 — Configure the KEDA ScaledObject

Create the `ScaledObject` resource (`keda-scaledobject.yaml`) to instruct KEDA to monitor the SQS queue and manage the HPA lifecycle:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: sqs-worker-scaledobject
  namespace: app-workload
spec:
  scaleTargetRef:
  apiVersion: apps/v1
    kind: Deployment
    name: sqs-worker
  minReplicaCount: 0
  maxReplicaCount: 10
  cooldownPeriod: 300
  pollingInterval: 15
  triggers:
  - type: aws-sqs-queue
    metadata:
      queueURL: 
      queueLength: "5"
      awsRegion: us-east-1
      identityOwner: pod

Apply the ScaledObject manifest:

kubectl apply -f keda-scaledobject.yaml

Production Hardening & Edge Cases

When running event-driven autoscaling at scale in production environments, several performance and concurrency nuances must be addressed:

• Polling Interval vs. API Rate Limits: Setting `pollingInterval` below 5 seconds can exhaust AWS SQS API rate limits (`ThrottlingException`), especially when managing multiple ScaledObjects. Maintain an interval between 15 and 30 seconds for standard non-latency-critical queues.
• Cooldown Periods and Scaling Flapping: The `cooldownPeriod` dictates how long KEDA waits after the last recorded event trigger drops to zero before scaling replicas back down to `minReplicaCount`. Set this value to align with your average job completion time to prevent premature termination of active worker threads.
• Pod Disruption Budgets (PDB): Because scaling down to zero terminates pods dynamically, ensure that critical stateful dependencies are handled outside worker containers. Avoid attaching PDBs with strict minAvailable constraints to scale-to-zero workloads, as this can block KEDA from scaling down replicas during low-traffic periods.
• IAM Least-Privilege Scoping: In production environments, restrict IAM policy resources to specific queue ARNs rather than wildcard (`*`) access.

Diagnostic Verification & Checklist

Use the following validation matrix to verify that infrastructure components, authentication mechanisms, and scaling triggers function correctly.

Monitor the KEDA metrics server logs for authentication failures using:

kubectl logs -n keda deployment/keda-metrics-apiserver --tail=100