On June 18, 2026, Anyscale and Google Cloud jointly published benchmark results showing a step-change in self-hosted LLM inference performance. Ray Serve LLM, working in tandem with vLLM, now delivers "up to 4.4x higher throughput on prefill-heavy workloads and 24x on decode-heavy workloads" compared to the previous Ray Serve LLM baseline, as reported by the vLLM project on X and documented in both companies' engineering blogs. If you run on-premise inference, or are evaluating whether to, these numbers move the economics meaningfully.
The optimizations behind the gains are already shipped: Ray 2.56 and current vLLM releases contain the changes. This isn't a roadmap item.
Three Optimizations, Three Bottlenecks Fixed
Each change targets a distinct pain point in the inference serving pipeline.
Direct Token Streaming
Previously, request routing and response streaming shared the same code path through the ingress router. Ray 2.56 separates them: tokens now stream directly from model replicas back to the client, bypassing the router on the return path. This cuts time-to-first-token (TTFT) and inter-token latency (ITL), particularly for long completions, document summarization, and chat-style workloads where streaming latency is directly user-visible.
v2 Ray Executor Backend for vLLM
This change moves Ray out of the inference data plane. Rather than Ray coordinating tensor operations, vLLM's native executor manages the engine loop while Ray handles orchestration and fault tolerance. The practical benefit is compounding: every engine-level improvement in vLLM, Flash Attention variants, paged KV-cache updates, speculative decoding, now applies automatically to Ray Serve deployments without waiting for a separate integration cycle. Previously, Ray Serve users often ran one or two vLLM versions behind because of integration lag.
HAProxy Integration
An integrated HAProxy layer handles load balancing across model replicas. This prevents the router's Python thread from saturating under high concurrency, a well-documented failure mode that causes latency spikes and dropped requests at production load. According to Google Cloud's blog, the three optimizations together deliver up to 5x higher throughput and 8x lower latency compared to the previous Ray Serve LLM configuration.
What This Means for Self-Hosted Deployments
The key metric for on-premise inference is throughput per GPU-hour: how many requests your fixed hardware can serve. When decode-heavy workloads achieve up to 24x higher throughput as reported in the Anyscale and Google Cloud benchmarks, three practical things follow:
The same hardware serves dramatically more requests. A setup handling 20 concurrent sessions now handles a multiple of that, without additional GPU spend. For organizations that have already invested in on-premise GPU infrastructure, this is a direct cost-per-query reduction.
The break-even point vs. cloud APIs shifts further in favor of on-premise. Cloud LLM APIs charge per token. On-premise carries fixed hardware costs. The higher the throughput, the more requests share those fixed costs, and the lower the effective cost per query. At 24x decode throughput improvement, the economics of self-hosting compound considerably at scale.
Data sovereignty stays intact. With Ray Serve + vLLM on-premise, nothing leaves your infrastructure. For legal, healthcare, finance, and HR workloads, any context where you process documents containing personal or confidential data, this is non-negotiable. No cloud API's performance numbers change that requirement. Learn more about what local AI means for data sovereignty.
Honest Scope of the Benchmark
The test hardware is enterprise-grade: NVIDIA HGX B200 GPUs on Google Cloud A4 virtual machines, eight replicas, Gemma 4 E2B as the test model (chosen to isolate orchestration bottlenecks from model inference). A single NVIDIA RTX 4090 or Mac Studio M3 Ultra won't hit these absolute numbers.
What does transfer across hardware sizes:
- The v2 executor backend and direct streaming are released features, not experimental code. They apply to any vLLM deployment, including single-GPU setups.
- HAProxy can be added manually in front of any Ollama or vLLM server without Kubernetes, standard reverse proxy configuration applies.
- Ray Serve on-premise (without GKE) benefits identically from Ray 2.56 improvements.
The percentage gains will vary by hardware and workload profile, but the architectural bottlenecks being addressed exist at every scale.
The Local LLM Stack in Context
When practitioners discuss self-hosted LLM deployments, the recurring question is whether operational overhead justifies the savings. These throughput gains shift that calculation. Serving more requests per GPU-hour means fewer GPUs for a given workload volume, which reduces both hardware acquisition cost and electricity consumption.
For a mid-size organization running 50,000 LLM requests per day internally, document analysis, drafting, internal knowledge retrieval, compliance checks, the difference between 5 tokens/s per request and 20+ tokens/s isn't just user experience. It's the difference between needing three A100s and needing twelve. Hardware costs at that scale compound quickly.
The performance gap between self-hosted and cloud-API inference has historically been the main argument for paying cloud providers. With successive improvements to Ollama, vLLM, and now Ray Serve LLM, that gap is narrowing on throughput while the data sovereignty and cost advantages of on-premise remain unchanged.
Freshlab helps organizations model this precisely before committing to hardware. Visit our local AI overview for the current landscape, or explore our pilot project program to test a specific workload on local inference before scaling.
What to Do Now
If you're already running vLLM in production:
- Update to the current vLLM release, the v2 Ray executor backend is included.
- Enable direct streaming if you're on Ray Serve (requires Ray 2.56+).
- Measure TTFT and ITL alongside requests/second, these are the metrics that reflect real user experience, not just raw throughput.
- Add HAProxy in front of your serving layer if you're experiencing concurrency-related latency spikes.
If you're not yet running local LLM inference and want to understand whether the numbers work for your organization's workload, reach out for a consultation. We'll map the throughput and cost figures to your actual use case.
Sources: vLLM Project on X · Google Cloud Blog · Anyscale Blog