Buoyant releases Kinference, an open source inference control plane for vLLM and Kubernetes

Buoyant announced Kinference, an Apache v2 open source control plane that routes OpenAI- and Anthropic-compatible inference traffic to vLLM workers on Kubernetes, adding token rate limiting, per-tenant usage and cost tracking, KV-cache-aware load balancing, and YAML inference policies. In the project's public benchmarks, Kinference cut median time-to-first-token by 95% and median end-to-end latency by 45% versus standard Kubernetes service load balancing on one workload, and served 7.4x the goodput on shared-prefix and about 3.5x on multiturn in a capacity test. Policies can route simpler prompts to cheaper models, burst overflow to providers such as Bedrock and Databricks, or keep PII-tagged requests inside the cluster, and per-tenant metrics cover token rates, TTFT, ITL and cached versus uncached prompt tokens.