Scaling Argo CD Across 34 Kubernetes Clusters With GitOps
Managing 34 Kubernetes clusters with one Argo CD instance shifts the hard problem from deploying applications to controlling change across a fleet. At Lucid Motors, a GitOps control plane spanned 22 AWS accounts, an Oracle Cloud tenancy, five regions and three continents. One repository held 174 Helm charts that rendered 1,214 Argo CD Application objects.
That scale makes each configuration shortcut expensive. The system depended on a carefully defined boundary between the central hub and the clusters it managed, a compact contract for describing each cluster, and continuous checks that showed reviewers the manifests a change would produce. The design also had to account for the failure of Git itself: on Oct. 7, 2026, a GitHub outage affected Actions, Git operations and pull requests, following smaller incidents on Oct. 5 and 6.
One hub, with authority bounded by each cluster
The central Argo CD instance managed platform components and held the cluster registrations, the Cluster API controller identity and three ApplicationSets. One ApplicationSet deployed a baseline to each cluster through an app-of-apps chart. Another placed shared platform services on the clusters designated to host them. The third managed the Cluster API definitions used to create clusters.
Each spoke cluster retained control over whether the hub could act inside it. Its own authentication configuration determined whether to accept the hub’s role. The spoke also owned the role used by Cluster API, while workloads received cloud roles through the identity provider configured on that cluster.
That division mattered especially in AWS. The hub held no long-lived cross-account secrets; it received short-lived credentials, and each spoke declared and reviewed its grants through Git. Oracle Cloud exposed a gap in the model: cluster registration still read a stored credential from a vault. That exception is an operational fact worth making explicit. A multi-cloud design can have a consistent interface while its security properties still differ by provider.
The practical lesson is to document those differences at the boundary. If one provider requires stored credentials while another supports short-lived access, operators need to know where the control plane’s assumptions stop applying.
A small cluster contract drives the fleet
The configuration model reduced each cluster to a file with roughly 20 keys. Those fields described identity, account boundary, region, environment, API access, delegated DNS and certificates, plus any opt-outs. The platform derived the rest from that record: which Applications to deploy, IAM role names, Cluster API identity, DNS delegation and cost tags.
This contract prevented teams from maintaining separate copies of related settings in multiple places. A new cluster joined a fleet through a label on its registration. A new component joined a fleet through a single list entry, with matrix generators handling the expansion across matching clusters.
The model also faced a useful test when the platform added an Oracle Cloud tenancy for a Middle East expansion. It entered through the existing cluster contract instead of requiring a bespoke path. That does not erase the credential exception, but it shows how a shared configuration model can accommodate a new account boundary without turning the fleet into a collection of one-off procedures.
Rendered output gives chart changes a fleet-wide review
With 174 charts, Argo CD itself wasn’t the main source of risk. A values change could render successfully for one cluster and fail across many others. The team addressed that risk by making CI render the configuration graph on every merge request.
Checks covered each chart on its own, every app-of-apps chart against every cluster contract, each cluster definition, and every chart referenced by an Application using the precise values that Application would receive. That last step connects template validation to actual deployment inputs, where incompatible assumptions can otherwise hide.
CI also rendered the target branch and attached the resulting manifest diff to the merge request. Reviewers could assess the Kubernetes resources that would change on the clusters instead of trying to infer effects from template edits alone. For a fleet this size, that moves review closer to the operational consequence of a change.
Drift data needs consistent definitions
Reported out-of-band configuration drift fell from 24.3 incidents a month before cutover to 0.9 afterward, using two 90-day windows. The comparison comes with a measurement caveat: the earlier figure was reconstructed from incident tickets tagged as drift, and tagging was incomplete. The later figure came from controller diff events with a confirmed out-of-band cause.
Those different collection methods don’t support a clean percentage reduction. They do show that, after drift instrumentation was in place, confirmed cases were uncommon and the controller corrected each one within its reconciliation interval without manual intervention. For teams adopting GitOps, the metric is most useful when the definition and collection method stay stable before and after a change.
Make destructive actions harder to trigger
The main failure mode at this scale was a bad commit with a wide reach. The team limited that risk by changing two deletion behaviors. Every ApplicationSet preserved deployed resources when removed, preventing deletion of an entry from cascading into resource removal. Cluster objects also carried a no-prune setting, so reverting a commit couldn’t delete a running cluster.
These safeguards reflect a straightforward operating principle: the effect of reconciliation should match the size and reversibility of the change. Removing a configuration entry and destroying a live cluster are not equivalent outcomes, so they shouldn’t share an easy default path.
GitHub’s October outage highlights the other side of the design. Applications continue running in the spokes if Git or the hub becomes unavailable, although GitOps-driven deployments and configuration updates stop until the control plane is available again. Keeping that interruption from becoming a workload outage is a useful measure of a hub-and-spoke architecture.
The hub also deserves treatment as a highly privileged system. If someone compromises a repository or CI pipeline and gets a change merged, the hub can carry that change across its reach. Repository controls, CI access and merge review therefore protect more than source files; they protect the fleet’s management plane.
Cluster onboarding became a repeatable operation
Adding a cluster followed four steps: create the IAM roles in Terraform using the naming pattern, write the cluster values file, open a merge request for CI to render the graph and publish its diff, then merge and label the registration. Older, manually built clusters could be adopted in place through an explicit import path.
Mean onboarding time changed from 4.0 days across eight earlier clusters to 1.7 hours across eleven clusters after cutover. The first figure was reconstructed from change tickets; the second measured the interval from merge to first successful sync. The cutover also removed manual approvals, and the measurement windows differed, so the figures aren’t a controlled comparison. The samples are small, but the process became markedly faster.
That reduction came from making the expected work visible and repeatable: define the cluster once, render its effects before merge, then let the controller reconcile it. At 34 clusters, the payoff comes from consistency across identity, deployment and review—not from adding more moving parts.
Scale raises the price of every shortcut
The underlying GitOps loop remains familiar: declare the intended state, review changes and let a controller converge on it. What changes with fleet size is the cost of weak defaults. A credential stored centrally, an unrendered values file or an unchecked prune setting may go unnoticed in a small environment. Across dozens of clusters, each can become a fleet-wide incident.
For platform teams, this case points to a practical design target: keep cluster configuration compact, make permissions originate at the spoke boundary, render the complete deployment graph before merge, and ensure outages pause change delivery without stopping running workloads. Those choices make the control plane easier to expand while keeping its failure modes legible.
Nitish Mane, now a software engineer at Philo working on infrastructure for AI-assisted production operations, previously worked on platform infrastructure at Lucid Motors, including this GitOps control plane. His account suggests that the next test for a fleet isn’t simply whether it can add cluster 35; it’s whether that cluster can join under the same contract and safety rules as the first 34.
