llm-d

Today at PyTorch Conference 2026, we announced llm-d 1.0! 🎉 Distributed inference is too complex to solve behind closed doors. 17 months in, llm-d is now a @CloudNativeFdn project powering fleets worldwide. Here’s what’s inside 1.0 🧵👇

During the PyTorch Conf keynote, Abdullah Gharaibeh gave a live demo of a couple of llm-d features. Watch live as llm-d routing and flow control provides real-time benefits for the user at scale.

RT @PyTorch: Live from #PyTorchCon 2026, Carlos Costa (@IBM, Abdullah Gharaibeh (@googlecloud), and Stephen Watt (@RedHat) present “Open Source Inference at Production Scale,” with a live @_llm_d_ demo underway.

🔒 Stable v1 API: A versioned configuration surface and hardened Endpoint Picker designed to give platform teams a dependable foundation for years to come. No breaking surprises—just reliable, production-ready infrastructure.

To everyone who benchmarked a model, reviewed a PR, or ran early builds: thank you. You proved the future of inference is open. 📖 Read the 1.0 deep dive: ⭐ Star on GitHub:

Great open-source communities push enterprise hardware further. 🤝 New work from IBM Research & Red Hat on llm-d: • 753B open model on 544 NVIDIA H100 GPUs • 5–10x lower cost per token vs commercial APIs • Serves 1000s of concurrent agents Blog:

RT @RedHat_AI: This is awesome. Claude Code and Codex running on open models with @vllm_project, no OpenAI API in the loop. Ask Qwen3.8 about a model released after its training cutoff and it has no idea. Give it server-side web search through the Agentic API and it finds the answer, with citations. The Agentic API keeps the stateful parts, turn history and tool execution, outside vLLM so the engine stays a stateless inference server. Built in Rust, MVP just landed. From @franciscojarceo (@RedHat_AI) and @Rxday000 (@EmbeddedLLM) 👇

RT @RedHat_AI: Every vLLM Office Hours runs with a live chat going the whole time. People ask real questions and get real answers, often from another attendee before a maintainer can get to it. Three from this week 🧵 1. Someone asked about the decode context parallelism tradeoff.

RT @RedHat_AI: vLLM Office Hours #56 recording is up: - What's new in vLLM 0.27 - Running Codex and Claude Code CLIs on open models - Speculators v0.6 and v0.7 update - GuideLLM v0.7.0: benchmarking reasoning, tool calling, and agentic traces Watch + see slides:

RT @redhatopen: AI keeps evolving: new models, hardware, & deployment patterns are reshaping how orgs build intelligent apps. @_llm_d_ shows what resilient AI infrastructure can look like. Spoiler: it’s not owned by one company. It’s strengthened by community.

RT @agicafet: 🔥 SPEAKER ANNOUNCEMENT | @vllm_project Bangkok Day (Thailand's largest event dedicated to vLLM) It is a real honor to announce this one. As the organizer, I'm proud to introduce another speaker I'm genuinely excited about: Jing Wen Ng Specialist Solutions Architect, AI, Asia Pacific @RedHat_AI And here's what makes it special: Jing Wen is flying to Bangkok to join us onsite, not virtually. 🎙️ Session: Scaling Distributed Inference with @_llm_d_ Red Hat is one of the teams behind llm-d, the project that takes vLLM beyond a single server and makes it scale across GPUs and nodes in production. What you'll take home: - What llm-d actually is, and why distributed inference is the next step once one vLLM instance stops being enough - KV-cache aware routing and prefill/decode disaggregation, and what they do to your serving performance - Running it for real on OpenShift AI, from single node to multi node deployments - Hard-won lessons from enterprise customers across APAC in FSI, telco, healthcare and the public sector on what moves AI from experimentation to production Why this session matters for Thailand: With registrations now past 500, the two topics our community asked for most were KV-Cache and real production use cases. This session sits right on top of both, delivered by someone who does this with enterprise customers across the region every week. 📅 Sunday, 6 September 2026, 2:00 PM to 7:00 PM 📍 JAS Green Village Ramkhamhaeng, 3rd floor (JAI Arena), Bangkok 🎟️ Free to attend. All sessions delivered in English. Huge thanks to our partners @inferact , @EmbeddedLLM, @RedHat_AI, @tensormesh , and @agicafet for making the largest AI Inference event in Thailand happen. "One vLLM Always vLLM" #vLLMBangkok #vLLM #llmd #RedHat #AIInfrastructure #LLMServing #KVCache #Inference #AGICAFET

RT @TerryTangYuan: 📢 𝗧𝗵𝗲 𝗦𝘁𝗮𝘁𝗲 𝗼𝗳 𝗠𝗼𝗱𝗲𝗹 𝗦𝗲𝗿𝘃𝗶𝗻𝗴 𝗖𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝗶𝗲𝘀: 𝗔𝘂𝗴𝘂𝘀𝘁 𝗘𝗱𝗶𝘁𝗶𝗼𝗻 𝗶𝘀 𝗼𝘂𝘁! Our goal with this newsletter is to give a clear, community-driven view of what’s happening across the model serving ecosystem that our teams at @RedHat_AI are contributing to, including updates from projects like @vllm_project, KServe, @_llm_d_, @kubernetesio, and beyond. 👉 Check out the August newsletter here: 👉 Subscribe to get future issues in your inbox: 🚀 Thanks to everyone who subscribed so far! Kudos to all contributors to this edition! David Simmons, Francisco Arceo, Jooho Lee, Yuan Tang, Wentao Ye, Sasa Zelenovic, Vivek Karunai Kiri Ragavan

RT @alexa_griffith_: Inside the HTTPRoute and how it works within @_llm_d_

RT @alexa_griffith_: Red Hat AI blog series part 3 is out now! This part focuses on how Red Hat's AI Inference on managed Kubernetes uses llm-d within the inference platform. @RedHat_AI @_llm_d_

RT @redhatopen: Open source is helping change the economics of #AI. Projects like @_llm_d_ are bringing open, collaborative innovation to LLM inference, helping organizations get more from the infrastructure they already have. Learn how: #OpenSource #AI #GenerativeAI

AI is entering the era of large-scale, distributed, agentic systems. Applications coordinate multiple models, tools, and services, process millions of requests, and demand enormous amounts of computer capacity.