Simon Mo✓

Woosuk Kwon @woosuk_k

Today, we're proud to announce @inferact, a startup founded by creators and core maintainers of @vllm_project, the most popular open-source LLM inference engine. Our mission is to grow vLLM as the world's AI inference engine and accelerate AI progress by making inference cheaper and faster. The Challenge Inference is not solved. It's getting harder. Models grow larger. New architectures proliferate: mixture-of-experts, multimodal, agentic. Every breakthrough demands new infrastructure. Meanwhile, hardware fragments: more accelerators, more programming models, and more combinations to optimize. The capability gap between models and the systems that serve them is widening. Left this way, the most capable models remain bottlenecked and with full scope of their capabilities accessible only to those who can build custom infrastructure. Close the gap, and we unlock new possibilities. And the problem is growing. Inference is shifting from a fraction of compute to the majority: test-time compute, RL training loops, synthetic data. We see a future where serving AI becomes effortless. Today, deploying a frontier model at scale requires a dedicated infrastructure team. Tomorrow, it should be as simple as spinning up a serverless database. The complexity doesn't disappear; it gets absorbed into the infrastructure we're building. Why Us vLLM sits at the intersection of models and hardware: a position that took years to build. When model vendors ship new architectures, they work with us to ensure day-zero support. When hardware vendors develop new silicon, they integrate with vLLM. When teams deploy at scale, they run vLLM, from frontier labs to hyperscalers to startups serving millions of users. Today, vLLM supports 500+ model architectures, runs on 200+ accelerator types, and powers inference at global scale. This ecosystem, built with 2,000+ contributors, is our foundation. We've been stewards of this engine since its first commit. We know it inside out. We deployed it at frontier scale—in research and in production. Open Source vLLM was built in the open. That's not changing. Inferact exists to supercharge vLLM adoption. The optimizations we develop flow back to the community. We plan to push vLLM's performance further, deepen support for emerging model architectures, and expand coverage across frontier hardware. The AI industry needs inference infrastructure that isn't locked behind proprietary walls. Join Us Through the open source community, we are fortunate to work with some of the best people we know. For @inferact, we're hiring engineers and researchers to work at the frontier of inference, where models meet hardware at scale. Come build with us. We're fortunate to be supported by investors who share our vision, including @a16z and @lightspeedvp who led our $150M seed, as well as @sequoia, @AltimeterCap, @Redpoint, @ZhenFund, The House Fund, @strikervp, @LaudeVentures, and @databricks. - @woosuk_k, @simon_mo_, @KaichaoYou, @rogerw0108, @istoica05 and the rest of the founding team

Stu Cao @caoshuxun

🚀 0 → 120 BILLION tokens/day in just ONE month! Just one month since onboarding Prime Inference to OpenRouter, we've scaled from zero to 120B tokens served daily. Note this is just a small portion of Prime Inference 🔥

Eric Zhang @ekzhang1

Alright I finally have more to share on my personal project! Resin is a runtime for building fast LLM inference engines. The idea is to make all the generic parts of inference reusable, and leave only the model-specific forward pass + kernels to be written for each architecture / hardware target. To build an engine: 1. Import Resin's core. 2. Have an agent write a forward pass for the model you’re interested in, for your specific hardware (GPU or TPU or AMD), any kernels you want. Then, boom, you have a standalone inference engine. — I've been hacking over the past month (early post: and recently this week I got a lot of features + parallelisms working, inspired by DeepInfer. For DeepSeek-V4.1-Flash, it's nearly as fast as DeepInfer (within 5-10%), while being self-contained and supporting other models, plus metrics + HTTP server + prefix cache. Yesterday I wrote a skill for authoring models and put Codex to work on it overnight. The agent built a GLM-5.3-Flash engine with MTP, reaching up to 60,000 input tok/s and 16,000 output tok/s on B200 (EP4). So far inference "vibe coding" has not been very reliable or successful. I think it's because people start from scratch each time and reinvent lots of tricky things on the scheduler and server layers. But with a shared core, we can automate lots of inference work. The tricky bit is writing a fast forward pass. But Resin turns that into a standalone engine by just importing a library, and it becomes a verifiable goal for the AI. Code:

Kevin Luu @khluu000

@yvbbrjdr FIFY

vLLM @vllm_project

vLLM matches DeepInfer, DeepSeek's own engine for DeepSeek-V4.1-Flash, on DeepSeek's decode configuration across batch sizes 8 to 256, and does it at DEP32 against DeepInfer’s EP64DP64. DeepInfer was originally built on vLLM. By integrating DeepSeek's open-source kernels and extensively optimizing prefill and decode, vLLM now matches DeepInfer's performance curves.

vLLM @vllm_project

Scaled out with P/D disaggregation and Wide-EP on a GB300 NVL72 rack, vLLM serves V4.1-Flash on SemiAnalysis AgentX at 954K tokens/s per GPU with 196 tokens/s P90 per user, 7.9× our single-node result. Our engineering blog covers the work behind it:

Kirill Bespalov @kbesplv

DeepSeek’s DeepInfer vs vLLM on one 8×B300 node: #deepinfer #deepseek #vllm

AGTP @AGTPinsights

VLLM and SGLang both just said they match DeepSeek's new DeepInfer engine. Here's what you need to know. DeepSeek open-sourced DeepInfer on Oct 20 under Apache 2.0. It is the in-house inference engine behind DeepSeek-V4.1-Flash, and it supports NVIDIA GPUs and Huawei chips. It was originally built on vLLM. Now the two big open serving engines are claiming parity with it. vLLM posted first on Oct 25. It says it matches DeepInfer on DeepSeek's decode configuration across batch sizes 8 to 256. It does that at DEP32, against DeepInfer's EP64DP64. vLLM did it by integrating DeepSeek's open-source kernels and heavily optimizing prefill and decode. The work was led by the Inferact team, with help from NVIDIA and SemiAnalysis' AgentX benchmark. SGLang says it matches DeepInfer on both prefill and decode. It first brought in DeepSeek's Mega kernels (MegaMoE, MegaAttention, Mega-mHC, MegaGate), SWA bounded replay and Engram optimizations. Then it added more kernel fusions and breakable CUDA graphs for prefill. Everything is in SGLang main, with an end-to-end reproduction guide. SGLang also points out gaps in DeepInfer's release. It leaves out a KV cache system, prefill/decode disaggregation and load balancing. SGLang ships all three. Its team also says some prefill comparisons mix GB300 results with DeepInfer's published B300 numbers, and argues that makes a fair comparison hard. Key numbers: - vLLM decode: matches DeepInfer at batch sizes 8 to 256, using DEP32 vs EP64DP64 - vLLM on a GB300 NVL72 rack: 954K tokens/s per GPU, 196 tokens/s P90 per user, 7.9x its single-node result - SGLang prefill: up to 1.04x throughput on 4x GB300 and 4x B300 - SGLang decode: up to 7.4% more throughput on 64x GB300 The result is that DeepSeek's release set a performance bar, and the two main open engines reached it within days.

vLLM @vllm_project

vLLM matches DeepInfer, DeepSeek's own engine for DeepSeek-V4.1-Flash, on DeepSeek's decode configuration across batch sizes 8 to 256, and does it at DEP32 against DeepInfer’s EP64DP64. DeepInfer was originally built on vLLM. By integrating DeepSeek's open-source kernels and extensively optimizing prefill and decode, vLLM now matches DeepInfer's performance curves.

vLLM @vllm_project

Scaled out with P/D disaggregation and Wide-EP on a GB300 NVL72 rack, vLLM serves V4.1-Flash on SemiAnalysis AgentX at 954K tokens/s per GPU with 196 tokens/s P90 per user, 7.9× our single-node result. Our engineering blog covers the work behind it:

vLLM @vllm_project

vLLM matches DeepInfer, DeepSeek's own engine for DeepSeek-V4.1-Flash, on DeepSeek's decode configuration across batch sizes 8 to 256, and does it at DEP32 against DeepInfer’s EP64DP64. DeepInfer was originally built on vLLM. By integrating DeepSeek's open-source kernels and extensively optimizing prefill and decode, vLLM now matches DeepInfer's performance curves.

vLLM @vllm_project

vLLM matches DeepInfer, DeepSeek's own engine for DeepSeek-V4.1-Flash, on DeepSeek's decode configuration across batch sizes 8 to 256, and does it at DEP32 against DeepInfer’s EP64DP64. DeepInfer was originally built on vLLM. By integrating DeepSeek's open-source kernels and extensively optimizing prefill and decode, vLLM now matches DeepInfer's performance curves.

vLLM @vllm_project

vLLM matches DeepInfer, DeepSeek's own engine for DeepSeek-V4.1-Flash, on DeepSeek's decode configuration across batch sizes 8 to 256, and does it at DEP32 against DeepInfer’s EP64DP64. DeepInfer was originally built on vLLM. By integrating DeepSeek's open-source kernels and extensively optimizing prefill and decode, vLLM now matches DeepInfer's performance curves.

vLLM @vllm_project

🎉 MiniMax M3.1 Flash from @MiniMax_AI has Day-0 support in vLLM! Built for multi-step automation, real-codebase engineering and visual reasoning—with native NVFP4 KV, NVFP4 MoE weights and DSpark speculative decoding in the serving stack. What the model can do, and the vLLM engineering that makes it practical to serve. 👇 Model: 1/5

Google Cloud @googlecloud

Native PyTorch on TPUs is here—and TorchTPU is officially open source. Train and serve models using the tools you already know: @vllm_project, @sgl_project, @_llm_d_, TorchTitan, @huggingface, and @raydistributed. Learn more from #PyTorchConf2026 →

Richard Socher @RichardSocher

Introducing Flint, @Recursive_SI’s first model: a fast local coding model built with our automated research system, the Eureka Machine. • Best-in-class fast local coding • Up to 2x faster than comparable models: 125+ tokens/sec on a MacBook Pro, 150+ on RTX Spark • Built from scratch in under 4 months Private, unmetered intelligence on your own machine. Open weights available now on @huggingface, run it locally on your favorite framework or try it for free on @opencode or @pidotdev.