Paying twice to pull the same image

A NAT Gateway is one of those resources that looks cheap on the pricing page and is not cheap on the bill.

The hourly charge is $0.045, about $33 a month. Nobody argues with that. The part that grows is $0.045 per GB processed — charged on every byte your private-subnet workloads send through it, in either direction.

For a Kubernetes cluster, the biggest consumer of that is usually not your application traffic. It is pulling the same container images, over and over, from a registry that lives in the same region.

The arithmetic

Take a fairly ordinary setup: nodes in private subnets, images in ECR, a cluster that scales through the day.

A 1.5 GB image — not unusual once you have a JVM or a Python ML stack in there — pulled onto 40 new nodes a day is 60 GB through the NAT Gateway. That is $2.70 a day, or about $80 a month, to move bytes between two AWS services in the same region.

Now make it realistic. Several services, each with its own image, nodes cycling on spot interruptions, a CI pipeline pulling base images, and DaemonSets landing on every new node:

A terabyte a day sounds like a lot until you count a 2 GB image landing on 500 node-starts. That is $16,000 a year to fetch bytes from a registry a few milliseconds away.

And none of it appears under "EKS" or "ECR" on your bill. It lands under NatGateway-Bytes.

Why it goes through NAT at all

A node in a private subnet has no route to the internet. To reach ECR it uses the default route, which points at the NAT Gateway. ECR's API and its storage layer are public endpoints, so as far as routing is concerned, pulling an image is internet traffic.

This is why the fix is not "cache better" — though caching helps. The fix is to stop the traffic leaving your VPC.

The endpoints that stop it

ECR needs three things, and one of them is free:

The S3 one matters more than people expect. ECR stores image layers in S3, so the bulk of what you transfer during a pull is S3 traffic, not ECR API traffic. A gateway endpoint for S3 carries no hourly charge and no per-GB charge at all.

That means the majority of your image-pull bytes can go from $0.045/GB to zero, and the remaining API chatter drops from $0.045/GB to $0.01/GB.

In Terraform:

# Layers live in S3 — this is the one that moves the most bytes, and it is free.
resource "aws_vpc_endpoint" "s3" {
  vpc_id            = var.vpc_id
  service_name      = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = var.private_route_table_ids
}
 
resource "aws_vpc_endpoint" "ecr_dkr" {
  vpc_id              = var.vpc_id
  service_name        = "com.amazonaws.${var.region}.ecr.dkr"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = var.private_subnet_ids
  security_group_ids  = [aws_security_group.endpoints.id]
  private_dns_enabled = true
}
 
resource "aws_vpc_endpoint" "ecr_api" {
  vpc_id              = var.vpc_id
  service_name        = "com.amazonaws.${var.region}.ecr.api"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = var.private_subnet_ids
  security_group_ids  = [aws_security_group.endpoints.id]
  private_dns_enabled = true
}

Two details that cause silent failures:

• private_dns_enabled = true is what makes it work without changing anything in your cluster. It overrides DNS inside the VPC so the normal ECR hostname resolves to the endpoint. Without it, pulls keep going out through NAT and you will see no error — just no saving.
• The security group on interface endpoints must allow 443 inbound from your node subnets. Miss this and pulls hang rather than fail cleanly, which is a confusing afternoon.

Do the interface endpoints pay for themselves?

Interface endpoints are not free, so this is worth checking rather than assuming.

Each one costs $0.01/AZ/hr. Across three AZs that is $0.03/hr, roughly $22 a month per endpoint. Two ECR endpoints is about $44 a month in fixed cost.

You recover that as soon as you are pushing more than ~1 TB a month through NAT for ECR — and remember the S3 gateway endpoint, which handles the bulk of the bytes, costs nothing and pays off immediately.

If your cluster is small and pulls little, add the S3 gateway endpoint anyway (free, no downside) and skip the interface endpoints until the volume justifies them.

While you are in there

The same logic applies to every AWS service your pods call from a private subnet. The usual suspects:

• logs — CloudWatch Logs, if you ship container logs
• sts — every IRSA token exchange
• ec2 and autoscaling — the cluster autoscaler or Karpenter
• secretsmanager or ssm — config and secret fetches

None of these move image-sized volumes, but they are constant, and sts in particular is called far more often than people realise.

How to see the number for yourself

In Cost Explorer, group by Usage Type rather than by Service, and look for NatGateway-Bytes. That single row is the one this article is about — and it is invisible when you group by service, because it hides inside EC2.

For attribution, VPC Flow Logs will tell you which workloads are actually driving it:

fields @timestamp, srcAddr, dstAddr, bytes
| filter dstAddr not like /^10\./
| stats sum(bytes) as total by srcAddr
| sort total desc
| limit 20

The top rows are your heaviest egress sources. Usually it is a handful of nodes pulling the same large images, which tells you whether to reach for endpoints, smaller images, or both.

What I would do differently

I have added VPC endpoints as a cost fix after the fact more than once. It works, but it is the wrong time to do it — you are changing network paths in a live cluster to recover money you already spent.

Endpoints belong in the landing zone, created with the VPC, before any workload runs. The S3 gateway endpoint in particular has no cost and no real downside; there is no version of this where you regret having it.

Put it in the module that creates private subnets, and the question never comes up again.

References

• Amazon VPC pricing — NAT Gateway hourly and data processing rates
• AWS PrivateLink pricing — interface endpoint hourly and per-GB charges
• Gateway endpoints for Amazon S3 — gateway endpoints carry no additional charge