vLLM vs TGI vs KServe: Choosing an LLM Inference Server on Kubernetes

Most posts frame this as a level three-way comparison. That framing is wrong. In August 2026, the honest picture is:

• TGI was archived on March 21, 2026 and is in maintenance-mode-only (Hugging Face announcement). New production deployments should not pick it.
• KServe is a serving *platform*, not an inference engine. It runs vLLM (or other engines) as its runtime. Comparing "KServe vs vLLM" is a category error.
• vLLM is the de-facto reference implementation. It is hosted by the PyTorch Foundation as of May 7, 2025, and was selected as the reference implementation for the MLPerf Inference 5.1 small-LLM benchmark in September 2025.

So the real question is not "which of these three?" It is: do you need an inference engine (vLLM) directly, or do you need a full Kubernetes serving platform (KServe) with vLLM inside it? This post answers that, then documents the migration path off TGI.

Layer diagram: where each project sits

`

Layer 3 Serving platform / K8s operator KServe (LLMInferenceService CRD)

Layer 2 Distributed inference control llm-d (CNCF Sandbox, runs vLLM)

Layer 1 Inference engine vLLM | TGI (archived) | SGLang | TensorRT-LLM

`

• Layer 1 (engine) is what actually runs the model on GPUs.
• Layer 2 (control) coordinates multi-node inference: KV-cache-aware routing, prefill/decode disaggregation, expert-parallel MoE serving. llm-d is the CNCF Sandbox project that provides this, founded in May 2025 by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA.
• Layer 3 (platform) is Kubernetes-native lifecycle management: CRDs, GitOps, autoscaling, canary rollouts. KServe owns this, accepted as CNCF Incubating on September 29, 2025.

Once you see the layers, the choice becomes concrete. Small deployment on one node? Run the engine (vLLM) directly. Enterprise Kubernetes shop with dozens of models, teams, and traffic patterns? Run KServe on top of vLLM.

What each project actually is, in 2026

vLLM

Open-source LLM inference engine originally from UC Berkeley's Sky Computing Lab, best known for introducing PagedAttention (SOSP '23). Apache 2.0. As of August 2026: 88.5k GitHub stars, 20.4k forks, 2,000+ contributors, latest release v0.21.0 (May 15, 2026), multiple releases per week. PyTorch Foundation hosted since May 7, 2025. Ships an OpenAI-compatible HTTP server plus Anthropic Messages API and gRPC endpoints.

Signal that matters most: MLCommons picked vLLM as the reference implementation for the new Llama3.1-8B benchmark in MLPerf Inference 5.1, explicitly citing its breadth across AMD, Google, Intel, and NVIDIA hardware. When the industry's neutral benchmarking body picks your project as the reference, that is community leverage no vendor blog can buy.

TGI (Text Generation Inference)

Hugging Face's production LLM server. Rust + Python + gRPC. Historically strong on streaming; used in production behind Hugging Chat, HF Inference API, and Inference Endpoints.

Current status: archived. On December 11, 2025 Hugging Face announced TGI was moving to maintenance mode, and the GitHub repo was made read-only on March 21, 2026. Hugging Face itself now recommends migrating to vLLM or SGLang for new deployments.

Note on licensing history: TGI briefly switched to the restrictive HFOIL 1.0 in 2023 before reverting to Apache 2.0 in April 2024 after community pushback. All shipping versions are Apache 2.0.

KServe

Kubernetes-native model serving platform. Rebranded from KFServing in September 2022, donated to LF AI & Data in February 2022, and accepted as a CNCF Incubating project on September 29, 2025. Apache 2.0.

KServe provides two CRDs:

• [`InferenceService`](https://kserve.github.io/website/docs/model-serving/getting-started/first-isvc) (v1beta1) . the classic predictive-AI abstraction. Wraps scikit-learn, XGBoost, PyTorch, TensorFlow, ONNX, Hugging Face, and vLLM runtimes on Knative or raw Kubernetes.
• [`LLMInferenceService`](https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview) (v1alpha1 as of KServe v0.16) . GenAI-first CRD built on llm-d. Supports single-node, multi-node distributed inference (via LeaderWorkerSet), disaggregated prefill/decode, and data-parallel / expert-parallel deployment for MoE models.

Since KServe v0.20 (August 2026), vLLM is a first-class supported runtime, and the LLMInferenceService CRD is explicitly built on the llm-d framework. That is not incidental. It is the current officially supported way to run production LLM inference on Kubernetes at scale.

Production users listed on the CNCF project page: Bloomberg, Red Hat, Cloudera, CyberAgent, Nutanix, SAP, NVIDIA.

2026 feature matrix

Benchmarks: what to believe (and what to distrust)

There is no single 2026 apples-to-apples benchmark across all three. Different benchmarks reach different conclusions. Report both, and be honest about the caveats:

• [BentoML llm-inference-benchmark](https://www.bentoml.com/blog/benchmarking-llm-inference-backends) (June 2024, Llama 3 70B Q4, A100 80GB, 100 concurrent users): LMDeploy and TensorRT-LLM led on raw decoding (~700 tok/s), TGI at 600-650, vLLM lowest TTFT but lower throughput at 4-bit quant. Caveat: predates vLLM v1 rewrite and TGI's archival.
• Practitioner benchmark, 2026 (Gigagpu, RTX 6000 Pro 96GB, Llama 3 70B, 64 concurrent): vLLM 4,200 tok/s vs TGI 2,900 tok/s (~1.45x). Vendor-adjacent, directional only.
• SGLang competitive pressure, 2025-2026 (MarkTechPost roundup): SGLang up to 6.4x higher throughput on structured workloads. On some H100 tests SGLang and LMDeploy each show a ~29% throughput advantage over fully-optimized vLLM. Important context: vLLM is not the raw-throughput leader in every scenario, but its breadth, community, and platform integration make it the safe platform bet.
• [MLPerf Inference v5.1](https://mlcommons.org/2025/09/small-llm-inference-5-1/) (September 2025): the new Llama3.1-8B reference implementation is vLLM. This is the closest thing to a neutral industry signal.

Two claims we deliberately do not repeat because we could not verify their primary source: "vLLM is 24x faster than TGI" (relayed by secondary sources citing a 2025 arXiv paper we could not confirm) and "TGI v3 is 13x faster than vLLM at 200k+ tokens" (Hugging Face blog claim, now historical since TGI is frozen and vLLM v1 added chunked prefill).

Deploying each on Kubernetes: the actual mechanics

vLLM Production Stack

The canonical way to run vLLM on Kubernetes is the [vLLM Production Stack](https://github.com/vllm-project/production-stack), a Berkeley + UChicago project first released January 22, 2025. Helm chart at https://vllm-project.github.io/production-stack. Requires vLLM 0.20+, Kubernetes 1.27+, and NVIDIA GPU Operator 24.9+.

Time to first token, roughly: 3-5 minutes to deploy the base stack, another ~10 minutes to wire up HPA and TLS ingress. Ships a router, cache servers, Grafana dashboards, model-aware and prefix-aware routing, KV-cache offloading, and multi-LoRA adapter serving. Full Helm deployment docs.

TGI on Kubernetes

No official Helm chart. Typical pattern is a Deployment + Service + Secret for HUGGING_FACE_HUB_TOKEN. The canonical reference is Hugging Face's GKE-TGI deployment guide. We do not recommend TGI for greenfield deployments given the archival. If you are already on TGI, the migration section below covers what to move to.

KServe with vLLM inside it

Install KServe via the operator, then apply a LLMInferenceService CRD pointing at vllm/vllm-openai:latest (or llm-d for multi-node inference). Autoscaling is native. GitOps-friendly. Canary rollouts and traffic splits are built into the CRD.

Two things worth knowing:

• ModelMesh (the older multi-model runtime) is being retired. The `kserve/modelmesh-serving` repository was archived on April 14, 2026, and Red Hat OpenShift AI 3.x is deprecating both ModelMesh and Serverless mode, migrating users to RawDeployment / Standard. If you deployed KServe before mid-2025 on ModelMesh, plan a migration.
• LLMInferenceService is v1alpha1 as of KServe v0.16. The API is still evolving. Do not build production automation that hard-codes the v1alpha1 group path if you can avoid it.

Choosing between them: the recommendation matrix

Migrating off TGI: what to change

Since TGI archival came up in the audits we ran through Q2 2026, here is the practical migration:

API surface. TGI's OpenAI-compatible Messages API maps directly onto vLLM's OpenAI-compatible server (--endpoint /v1/chat/completions). Most client code needs zero changes. Prompts, temperature, top-p, stop tokens, tool-calling schemas all transfer.

Container image. ghcr.io/huggingface/text-generation-inference:latest becomes vllm/vllm-openai:latest. Change the entrypoint from TGI's --model-id to vLLM's --model. Both accept Hugging Face model IDs directly.

Authentication for gated models. HUGGING_FACE_HUB_TOKEN becomes HF_TOKEN (vLLM standardized on the shorter env var).

Autoscaling. TGI worked with standard HPA on Kubernetes. vLLM does the same. Move your HPA manifest as-is, but consider layering KServe's LLMInferenceService on top if you want scale-to-zero and canary out of the box.

Multi-model routing. If you were using TGI + a custom router, the vLLM Production Stack ships one. If you already run KServe, use ModelMesh's replacement (Standard / RawDeployment mode).

Estimated migration effort: for a single-model production deployment, half a day if your CI/CD is clean. For a multi-model shop with custom routing and monitoring, plan 1-2 weeks including regression benchmarks.

What we run in client deployments

We deploy KServe on top of vLLM for every new production LLM inference engagement in 2026. Justifications:

1. The CRD abstraction is worth the operator overhead. Version rollouts, traffic splits, and scale-to-zero are one YAML change instead of one custom deployment pipeline.
2. vLLM is the safe engine bet. Community leverage, MLPerf reference status, and PyTorch Foundation governance make it the least likely to be orphaned mid-engagement.
3. KServe's LLMInferenceService gives us upgrade room. As llm-d matures and adds distributed multi-node inference, disaggregated prefill/decode, and better MoE serving, we get those features by upgrading the CRD, not by rewriting the platform.

For teams running production LLMs on their own Kubernetes clusters or on managed Kubernetes at hyperscalers, this is the current answer. Ask us again in 12 months and it may be different, because the pace of change in this space is faster than any single technology bet lasts. But in August 2026, the answer is: vLLM as the engine, KServe as the platform, plan the migration off TGI now.

Related eprecisio content

For how we deploy production Kubernetes for GPU workloads including this inference stack, see our on-prem GPU Kubernetes reference architecture and MLOps service. For the broader MLOps toolchain (Kubeflow Pipelines, MLflow, ArgoCD, Prometheus/Grafana), see our MLOps tools in production post.

If you are picking an LLM inference stack for a new deployment or evaluating a migration off TGI, book a free 30-minute call.