Kubernetes-Native ML Platforms Need Declarative Lifecycle Control
Running machine learning workloads on Kubernetes solves only part of the platform problem. Training jobs can use cluster scheduling, inference services can run as Deployments, and GPUs can be assigned across clusters. The harder question is whether teams can manage the full model lifecycle through the same declarative, observable operating model.
That distinction is becoming more relevant as AI workloads move onto Kubernetes, a trend described by Vara Bonthu in a recent CNCF article about AI platforms converging on Kubernetes. Kubernetes handles compute, isolation, scaling and recovery. Yet training, evaluation, model registration, deployment and rollback may still depend on separate APIs, databases and custom scripts. The cluster runs the workload; glue code manages what happens to it.
My read is that the platform opportunity lies in bringing those lifecycle operations under explicit control. Kubernetes offers a useful pattern: declare a desired state, observe the current state, then reconcile the gap. Applied to machine learning, that approach can make model operations easier to track, recover and automate across changing infrastructure.
Model delivery is a continuing state machine
A production model is connected to more than a running process. Its history can include the training run, code version, parameters, input dataset and evaluation results. A deployment then refers to a particular model version and carries its own rollout policy, health signals and traffic allocation.
Those connections need to persist through failures, process restarts and changes in the teams or tools responsible for the system. Treating each stage as an isolated job makes it harder to establish what produced a model, what is currently serving it, or whether a rollout met its conditions. At platform scale, the lifecycle behaves more like a state machine that needs continuous management than a fixed sequence of scripts.
That has practical consequences for reliability. If the platform records lifecycle state explicitly, a controller can retry after transient failures and report where progress stopped. Operators can distinguish a deployment waiting for capacity from one blocked by policy or an unhealthy model, instead of having to reconstruct the situation from scattered logs and service-specific status pages.
Reconciliation turns deployment intent into observable action
Consider a request to send 10% of production traffic to model version 42. The request describes the desired state. If the version currently receives no traffic, a controller can start a canary rollout, check health signals and update the resource’s status as it proceeds. If the rollout stalls, status conditions can show whether capacity, a policy check or model health is the obstacle.
- Declare the target: Version 42 should receive 10% of traffic and remain healthy.
- Check current state: The version is receiving 0% of traffic.
- Make the change: Create or update the resources that control the runtime.
- Report progress: Record conditions, failures and rollout status.
- Reconcile again: Continue until the target is reached or intervention is needed.
Across clusters, the same control pattern can apply changes gradually and stop a rollout when problems appear. Custom resources and controllers let platform teams express domain concepts such as training runs, evaluations and deployments while retaining Kubernetes’ reconciliation model.
The value here is not simply that an API can create a workload. It’s that the system can keep comparing intent with reality, explain its current state and retry as conditions change. That makes recovery part of ordinary operation rather than a collection of special-case scripts.
Make lifecycle state visible, keep execution specialized
When lifecycle concepts become explicit resources, their relationships can form an inspectable object graph. A training run can produce a model resource that records lineage and evaluation results. A deployment can point to that model and report health and traffic allocation. The resources provide a durable view of how work moves through the platform.
This can improve day-to-day operations in several ways. Shared status conditions give users a consistent vocabulary for states such as pending, progressing, ready, degraded or blocked. Admission controls can check fast, deterministic requirements, including resource limits and data restrictions; controllers can coordinate longer evaluation and approval steps. Versioned, typed APIs also give web interfaces, command-line tools, controllers and AI agents a common contract to build against.
Observability gains context when metrics and logs connect to lifecycle objects. “Pod failed” identifies a symptom. “Evaluation failed for model candidate X” tells an operator which part of the model process needs attention. That connection can shorten the work required to understand a failure and decide what to retry or roll back.
A control plane should coordinate specialized tools rather than absorb every capability into one platform. It can define lifecycle intent, relationships, policy and status. Training, data processing and inference engines can continue doing the compute-heavy work. Adapters translate the control plane’s intent into runtime-specific resources, then return results to the shared lifecycle model. This gives users a consistent operating experience while allowing platform teams to change execution engines without redesigning the entire interface.
Choose Kubernetes resources for durable intent
Not every internal detail belongs in the Kubernetes API. A useful lifecycle resource represents intent that users or automation need to manage over time, such as a model deployment. A temporary function call or a large evaluation artifact may not meet that bar.
Large models, detailed evaluation results, logs and traces can remain in stores designed for those data types. Kubernetes resources can hold durable intent, references and concise status. Before adding a resource, platform teams should ask whether it represents meaningful user intent, has a lifecycle beyond one process, needs status tracking and benefits from ongoing reconciliation. The API should also be stable enough to serve as a long-term contract.
This boundary matters because an API crowded with transient implementation details is harder to maintain and less useful to users. The aim is to make important lifecycle state visible and manageable, while leaving each execution system room to handle the details it knows best.
The lifecycle contract outlasts individual engines
Experience with Uber’s ML platform, Michelangelo AI, helped make the separation clear. Its modular control plane has been released as an open-source project. The lasting abstraction, in this view, is the lifecycle contract connecting training runs, models, evaluations and deployments—not a specific training engine, serving runtime or workflow implementation.
Those components change over time, and different workloads call for different tools. A consistent set of lifecycle resources and reconciliation patterns can let specialized runtimes evolve while preserving a familiar way to track and operate models.
The same operational questions extend to generative and agentic AI, where platforms may also track prompts, tools, policies, benchmark suites and traces. Teams still need to know what they asked a system to do, which versions and inputs it used, what happened, whether it met evaluation and policy requirements, and whether they can reproduce, promote or roll back the result. Those questions call for explicit state and traceable relationships, whatever the underlying artifacts become.
Shared patterns are the next platform task
Kubernetes has provided a common foundation for running AI workloads. The next opportunity is for the open-source community to establish useful patterns for managing the lifecycle around them. A single universal MLOps API is unlikely to fit every workload, but teams can still work toward shared conventions for lineage, resource status, portable workload definitions, identity and policy boundaries, evaluation and promotion, and visibility into compute, accelerators and storage.
The practical test for a Kubernetes-native ML platform is therefore what it can manage and explain after a workload starts. When training, evaluation and deployment are represented as durable state, teams can trace relationships, recover from interruptions and automate rollouts across tools and clusters. That operating model is the step that turns Kubernetes from a place to run AI into a foundation for managing its lifecycle.
