Rethinking Load Balancing: The Impacts of Cross-AZ Traffic in Kubernetes Service Meshes

Understanding the Sluggishness of “Just Works” Load Balancing

If you’re managing a multi-availability zone (AZ) Kubernetes cluster layered with a service mesh, there’s a strong likelihood you're incurring hidden costs that don’t appear on your typical operational dashboards.

The Undiscussed Default

Kubernetes Services, along with Istio’s Envoy sidecars, often default to a random traffic distribution across all healthy endpoints without considering the availability zones. This default setting might serve a single AZ deployment well, but as pods are distributed across three AZs for added resilience—effectively a standard practice—this randomness morphs into a problem. A pod situated in us-east-1a initiating a call to a downstream service faces around a two-thirds likelihood of landing on a pod located in a different AZ. Multiply this across three or four service interactions, including database access, and a single user request can incur various cross-AZ hops before producing a response.

The predicament intensifies with AWS’s cross-zone load balancing. By default, an AWS Network Load Balancer (NLB) in front of an Istio ingress gateway can route incoming requests to any AZ. This becomes particularly troubling when a healthy target resides close to the load balancer node handling that request.

The Financial and Latency Consequences

There are two main costs worth examining.

Increased Latency: A medium-sized production cluster processing approximately 3,500 requests per second (RPS) across three AZs shows that same-AZ requests result in a median latency (p50) of 15-18ms. In contrast, cross-AZ requests to the same service take around 25-30ms, demonstrating a slowdown of 40% to 65%. With about two-thirds of requests crossing AZs, the weighted median latency settles at around 24ms instead of the 17ms benchmark. This delay compounds exponentially with additional internal service interactions.

Unexpected Costs: AWS charges $0.01 per GB for data that traverses AZs within a region. Although this seems negligible on a per-request basis, internal service communication typically far exceeds external traffic volume, often ranging from 5x to 10x due to API calls, database reads, and other methodologies. In the examined environment, a conservative estimate for expenses related to ingress, service interactions, database access, and Kafka traffic approximated $600 monthly before corrective measures, excluding additional expenses for automatic Aurora cross-AZ replication and log shipping. Since actual costs depend heavily on your own traffic patterns, it’s advisable to check your figures in AWS Cost Explorer for more accurate insight.

A Solution for Load Balancing: Locality-Aware vs. Locality-Only

Istio offers a functionality for locality-aware load balancing via the DestinationRule traffic policy. The common approach might suggest directing 100% of traffic to the local AZ, but this isn’t advisable. Such a method hampers your ability to effectively manage failures and mitigates recovery from issues within a specific AZ.

Instead, a more balanced and effective strategy is a weighted distribution: direct around 80% of traffic to the same AZ, allocating the remaining 20% split equally between the other two AZs. This approach incorporates outlier detection, which automatically eliminates unhealthy endpoints.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: checkout-service-locality-lb
spec:
  host: checkout-service.prod.svc.cluster.local
  trafficPolicy:
    loadBalancer:
      localityLbSetting:
        enabled: true
        distribute:
          - from: us-east-1a/*
            to:
              "us-east-1a/*": 80
              "us-east-1b/*": 10
              "us-east-1c/*": 10
    outlierDetection:
      consecutiveErrors: 5
      interval: 30s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

Key Considerations for Successful Implementation

For this solution to function effectively, consider three critical factors that are often overlooked:

1. Even Distribution of Pods Across AZs: An AZ with 50% of the service’s pods will receive more than half of all traffic if the locality policy is set to 80/10/10. To ensure equitable distribution, utilize topologySpreadConstraints with a maxSkew setting of 1. Otherwise, the locality policy will merely shift the existing imbalance, rather than correcting it.
2. Ingress Gateway Pods Must Be Distributed Across AZs: Locality-aware routing at the service level will fall short if ingress pods are heavily concentrated in one AZ, as every request starts at those ingress points.
3. Disable NLB Cross-Zone Load Balancing: This can be achieved via the annotation service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "false". If skipped, the load balancer itself will undermine the changes made at the mesh layer.

Understanding Benefits and Trade-offs

While latency improves, the more profound impact is on the failure blast radius. Previously, under a random-distribution model, losing one AZ meant the other two AZs had to absorb a sudden increase in load, escalating the risk of saturation. With an 80/10/10 locality policy, only the traffic that the lost AZ was handling directly is redistributed among the others, minimizing the relative increase in load and allowing for smarter handling of outlier situations.

However, this improvement comes with increased operational complexity. Traffic distribution won’t follow the intuitive 33/33/33 split, meaning that those troubleshooting unexpected traffic patterns will need to understand the intricacies of the locality policy. Clear documentation is essential to circumvent confusion when anomalies arise.

Final Thoughts Before Deployment

Your first step should be to validate this setup in a staging environment with realistic loads using tools like k6 or Locust to expose any load imbalances resulting from uneven pod distribution. Consider canary testing on low-traffic services in production before adjusting any high-impact components, keeping a close eye on error rates and latency metrics.

Locality-aware load balancing isn’t a novel concept; it’s a feature within Istio that has existed for years, often overlooked merely because the mesh operates adequately without it—until the costs of cross-AZ traffic or failures become clear. Taking the time to adjust your approach can ensure your multi-AZ cluster functions as intended.