Where your EKS bill actually goes

Every few months someone forwards me a cost graph with a question attached: we did not change anything, why is this going up?

They are usually looking at the EC2 line, because that is the big one. Ninety percent of an EKS bill is compute, so that is where attention goes. But compute is also the line that gets watched, right-sized and committed to. It is rarely where the surprise is.

The surprise is almost always in a line item that has no owner, scales with traffic rather than with headcount, and does not appear on anybody's dashboard.

Here are the five I check first.

1. Falling behind on Kubernetes versions

This one is the cheapest to fix and the most expensive to ignore.

An EKS control plane costs $0.10 per cluster per hour on standard support — about $73 a month. Once your Kubernetes version leaves standard support and moves to extended support, that becomes $0.60 per cluster per hour.

That is roughly $438 a month per cluster, for exactly the same thing you had before.

Nobody decides to pay this. It happens because an upgrade got deferred, then deferred again, and the version quietly aged out. If you run three clusters — dev, staging, production — a deferred upgrade is over $1,000 a month for nothing.

Worth checking before you read any further:

aws eks describe-cluster --name my-cluster \
  --query 'cluster.{version:version,status:health.issues}'

If you are on extended support, the upgrade has already paid for itself.

2. Cross-AZ traffic between pods

Spreading across Availability Zones is correct for availability. It also means a pod in us-east-1a talking to a pod in us-east-1b generates billable traffic — $0.01 per GB leaving one AZ and $0.01 per GB entering the other.

The scheduler does not know or care which AZ your chatty services land in. A service mesh sidecar, a Redis client and an application pod scattered across three zones will happily bounce traffic between them all day.

This is invisible on a per-request basis and substantial in aggregate. A service doing a few hundred megabits of internal chatter can add four figures a month in transfer charges that appear under "EC2 — Data Transfer" rather than anywhere near your cluster.

The fix is usually topology-aware routing, so traffic prefers an endpoint in its own zone:

apiVersion: v1
kind: Service
metadata:
  name: payments
  annotations:
    service.kubernetes.io/topology-mode: Auto
spec:
  selector:
    app: payments
  ports:
    - port: 8080

You do not want this everywhere. It trades a little resilience for cost, so apply it to high-volume internal services and leave your front door alone.

3. NAT Gateway data processing

A NAT Gateway costs $0.045 per hour to exist, which nobody minds. It also charges $0.045 per GB processed, which is where it gets interesting.

Every image pull, every package download, every call to an external API from a pod in a private subnet goes through it and is billed per gigabyte.

The pattern I see most: a cluster pulling large container images on every scale-up event, through NAT, repeatedly. A 2 GB image pulled 500 times a day is 1 TB through the gateway — about $45 a day just to fetch bytes that live in ECR a few milliseconds away.

Two things fix most of it:

• VPC endpoints for ECR, S3 and the other AWS services your pods actually call. Traffic to them stops traversing NAT entirely. The S3 gateway endpoint in particular is free.
• Image hygiene — smaller base images, and node-level caching so scale-ups are not re-pulling the world.

4. Storage nobody deleted

Two habits accumulate cost quietly:

• Orphaned EBS volumes. A PersistentVolumeClaim with the default Delete reclaim policy cleans up after itself. One set to Retain does not. Every deleted StatefulSet that used Retain leaves its volumes behind, provisioned and billed, forever.
• Snapshots. Automated volume snapshots with no lifecycle policy grow without limit. They are cheap individually, which is exactly why nobody notices the total.

Finding the orphans takes one command:

aws ec2 describe-volumes \
  --filters Name=status,Values=available \
  --query 'Volumes[].{ID:VolumeId,GB:Size,Created:CreateTime}' \
  --output table

Everything available is a volume attached to nothing, being paid for. Check before deleting — "attached to nothing" and "safe to delete" are not the same sentence — but this list is usually longer than people expect.

5. Logs you will never read

CloudWatch Logs bills on ingestion and on storage. A cluster with debug logging left on after an incident, or an ingress controller logging every request body, ingests a surprising volume.

The part that catches people is retention. The default for a new log group is Never expire. Set once, forgotten, billed monthly, forever.

aws logs put-retention-policy \
  --log-group-name /aws/eks/my-cluster/cluster \
  --retention-in-days 30

Thirty days is a reasonable default for cluster logs. If you need a year of audit data, that belongs in S3 with a lifecycle policy, not in CloudWatch at CloudWatch prices.

How to actually find yours

Guessing is slower than looking. The blocker is almost always that the bill cannot be attributed, so start there:

1. Turn on cost allocation tags and enforce them. Without tagging you are guessing about which team or service owns what.
2. Split by usage type, not by service. In Cost Explorer, group by Usage Type rather than Service. NatGateway-Bytes and DataTransfer-Regional-Bytes are their own rows — they do not hide inside "EC2" any more.
3. Deploy OpenCost or Kubecost. AWS bills you per account; Kubernetes spends per namespace. You need something that maps one to the other, or you will never answer "which team caused this".
4. Set an anomaly alert. AWS Cost Anomaly Detection is free. A spike you hear about in three days is much cheaper than one you find at the end of the month.

What I would do differently

The mistake I have made, and seen made, is treating cost work as an audit — a project with a start and an end, where someone finds the waste, cuts it, and writes a summary.

It comes back. It always comes back, because the thing that created it was a default: a reclaim policy, a retention setting, a scheduler that does not know about zones. Nobody chose the cost, so removing it once changes nothing about the next deployment.

What holds is making the defaults right and making the bill visible to the people who move it. A per-namespace cost figure in a channel that engineers already read will do more than any one-off cleanup — mostly because it turns an invisible number into one somebody feels responsible for.

Start with the extended support check, though. That one is a single command and it might be $400 a month.

References

• Amazon EKS pricing — control plane rates for standard and extended support
• Amazon VPC pricing — NAT Gateway and inter-AZ data transfer
• Gateway endpoints for Amazon S3 — why the S3 endpoint carries no charge