Buoyant Kinference Brings Workload Controls to Kubernetes AI Inference

Why inference needs its own controls

GPU capacity is expensive, and AI inference workloads can leave it underused even when teams are competing for access. That mismatch is a sign that raw hardware availability doesn’t tell the whole story: a cluster can feel constrained to one team while GPUs sit idle at other moments or under other workloads. Buoyant’s open-source Kinference aims to address that gap with a control plane for managing vLLM workloads across Kubernetes clusters.

The practical idea is to give IT teams policy controls over inference alongside the tools that handle model requests themselves. Teams can set boundaries for performance, cost and compliance, then direct resources across workloads according to those priorities. That matters because model serving isn’t just a question of getting a model to respond; operators also have to decide which requests get capacity, how much service each workload receives and what happens when demand changes.

Kinference is intended to make self-hosted AI easier to operate as it joins established applications on Kubernetes. That’s a familiar operational setting for many organizations, but inference brings its own resource demands and competing service expectations. A control plane can provide a place to express those priorities. Whether it can enforce them effectively will depend on the behavior of real workloads.

Managing inference across a cluster

Kinference works with vLLM, an inference engine and server originally developed by researchers at the University of California, Berkeley’s Sky Computing Lab. vLLM uses PagedAttention to improve memory throughput, while its server manages network traffic. Kinference adds a layer for coordinating those workloads in distributed Kubernetes environments, where operators may need to manage requests and resources across more than one part of a cluster.

The distinction between serving requests and coordinating workloads is useful. An inference engine handles model execution and request serving; a control plane can apply policies around where workloads run and how resources are assigned. In practice, those layers need to work together. A policy that looks sensible on paper can still disappoint if it doesn’t reflect the demands of the models, applications and users sharing the infrastructure.

Buoyant CEO William Morgan said the control plane is intended to help teams scale AI workloads more cost-effectively. Instead of paying cloud providers for model access by token, organizations can assign infrastructure resources to workloads and manage the quality of service each receives. That shifts the management question toward the cost and use of owned or controlled capacity, though it also makes teams responsible for operating that capacity.

That approach may appeal to organizations handling sensitive data or facing latency demands. Keeping inference on infrastructure they control can reduce exposure to third-party services, while closer placement may suit applications that make frequent, iterative requests. Kinference lets operators define operational limits around those concerns, giving them a way to express priorities as policies rather than treating every request as interchangeable.

Making GPU capacity work harder

One of Kinference’s central jobs is coordinating workloads to improve GPU utilization. Morgan pointed to a mismatch in AI infrastructure: GPUs can be scarce, yet their utilization is often in the single digits. The numbers here are underwhelming, but the underlying problem is more useful than a utilization figure alone: operators need to understand whether idle capacity can serve another workload without undermining the service already running.

Kinference uses GPU cache data, queue status and workload controls to help distribute inference requests across available capacity. Those signals give operators information about both the hardware and the work waiting to use it. Similar resource-management systems typically have to balance competing goals: keep expensive equipment busy, preserve responsiveness and respect the limits assigned to different workloads. There’s no single setting that guarantees all three.

That coordination matters when several teams or applications share a cluster. Kinference can track GPU consumption by tenant, group or team, giving operators a way to see where capacity goes and attribute infrastructure costs. Those records could inform decisions about workload allocation and future capacity needs. They also make internal tradeoffs easier to discuss: a team’s demand can be considered alongside its resource use and the service expectations attached to its work.

And this is the part most people overlook: visibility doesn’t create capacity. It can help operators identify waste or make a better case for changing allocations, but the gains depend on what the workloads can share and how the policies behave under pressure. If you’re working in this space, the practical test is whether the controls help teams make decisions they can explain, then keep those decisions aligned with actual demand.

Keeping a path to cloud capacity

Self-hosting doesn’t have to mean keeping every request on premises. Operators can configure Kinference to send selected inference traffic to cloud providers when conditions warrant it, such as a saturated GPU pool, a particular tenant’s usage or characteristics of a request. That gives teams a way to extend capacity while retaining control over when requests leave their own environment.

This flexibility addresses a real operational tension. Local infrastructure can offer more direct control over placement and resource use, but demand may exceed what’s available at a given moment. Cloud capacity can provide another route for selected traffic, though deciding what should be sent out requires policies that match the organization’s performance, cost and compliance priorities. Here’s the thing: the routing choice is part of the product’s control problem, not just an emergency escape hatch.

That said, the source doesn’t establish how much capacity Kinference can shift, how quickly it can respond or what cloud routing costs in practice. Those questions depend on workload behavior and operator configuration. The useful point is that teams can set conditions for routing, rather than being limited to an all-local or all-cloud choice.

A control problem that will grow with AI

The number of AI workloads running on Kubernetes isn’t specified, but the operational challenge is already familiar: teams must fit a new class of applications into limited infrastructure shared with existing services. If AI workloads continue to grow, deciding how to allocate GPUs, preserve service quality and manage costs will become a larger part of cluster operations. This is more significant than it looks, because the day-to-day constraint may be coordination as much as hardware supply.

Kinference’s value will depend on how well its controls work with real workloads and how much additional utilization they produce. Its design points to a practical direction for self-hosted AI: manage inference as a shared, policy-driven infrastructure service, with cloud capacity available when local resources run short. A control plane can make that model easier to govern, but operators will still need to judge whether its policies deliver the service levels they expect.

For teams weighing self-hosted inference, the question is whether this kind of coordination solves a problem they already face: competing workloads, costly idle capacity or a need to control where requests run. The promise is clear. The proof will come from operating it against the varied demands of an actual cluster.