ALERT🚨: GB300 NVL72 @sgl_project can get close to 1000 tok/s/user on DeepSeek Flash v4.1 by aggressively fusing kernels together!!
TileRT started this trend in the ecosystem of megakernel engines on OSS frontier models! Glad to see SGLang widely adopting it!
Great work by the SGLang, @radixark, and NVIDIA teams on pushing performance!
ASE showcased its panel-level packaging (PLP) solution for AMD’s Venice CPU, with initial production targeted for Q1 2027, likely at limited scale at its Kaohsiung facility. ASE and its subsidiary SPIL have worked with AMD to implement the second generation of their Fan-Out Embedded Bridge (FOEB) technology in Venice. The current 310 × 310 mm panel accommodates 25 Venice interposers after edge loss, compared with 22 on a traditional round 300 mm carrier wafer, roughly a 14% increase in gross package count. This modest improvement in package output per carrier represents a starting point for panel-level interposers with Venice’s 2.7x reticle size, rather than evidence of a dramatic reduction in cost. (1/2)🧵
Source: OCP ASE Booth
The larger cost reduction opportunity lies further out with larger interposers; for example, 9.5x reticle size interposers yield 9 per panel versus 4 per wafer. The transition to larger reticle sizes and larger panels depends on equipment maturity. (2/2)
ALERT🚨: The number of tokens that can be cached can increase by 20-40%🚀 by enabling DRAM KVCache offloading optimization at 12-16 concurrent agentic sessions. A B200 only has 180 GB of HBM, and after weight storage, only 100 GB of HBM per chip remains for KVCache.
THIS IS NOT ENOUGH FOR LONG-CONTEXT AGENTIC INFERENCE WORKLOADS. Thus, with SGLang HiCache, you can also offload KVCache to the CPU's DDR5/LPDDR5X memory, which helps improve performance. (1/2)🧵
At a small number of concurrent sessions, CPU DRAM offloading is not needed since all the sessions' KVCache (even for idle sessions waiting for tool-use calls to finish) can be stored in HBM. (2/2)
We’re deepening our collaboration with @googlecloud to give customers more options for building and running AI at scale.
Google TPUs are coming to Oracle Cloud Infrastructure (OCI), marking the first time Google TPUs will be available on a cloud platform outside Google. Customers will have more flexibility to match AI infrastructure to demanding training and inference workloads, optimizing their environments for performance, throughput, and economics.
Learn more: #AIWorld
We’re deepening our collaboration with @googlecloud to give customers more options for building and running AI at scale.
Google TPUs are coming to Oracle Cloud Infrastructure (OCI), marking the first time Google TPUs will be available on a cloud platform outside Google. Customers will have more flexibility to match AI infrastructure to demanding training and inference workloads, optimizing their environments for performance, throughput, and economics.
Learn more: #AIWorld
Anthropic’s S-1 shows inference revenue per megawatt nearly quadrupling in a year.
“Inference revenue per megawatt went from sixteen million last year to sixty plus currently, across the entire fleet. AI model businesses scale extremely well.”
“Revenue grew 1,368% year over year for the three months ended June 30, compared to compute and infrastructure expenses that grew 300% during the same period.”
“How much of that is Anthropic being geniuses versus market-wide efficiency? We’re seeing GB be 50x better than Hopper.”
“They have TPU, Trainium, NVIDIA GPU and AMD GPU all online. New hardware is always more performant, but you need the software to use it. Who’s got the best software? Probably the company with the top engineers and the best model.”
RT @tbpn: The @SemiAnalysis_ team gives two examples why we're still so early with AI adoption outside of the tech bubble:
1. $1,500/month token budgets at Fortune 500 tech companies:
SemiAnalysis Tokenomics co-lead @SaasquatchC explains that a lot of engineers don't even hit that budget at these big, older companies, and that traditional back-office work is lagging behind, as well.
2. The "AI-pilled" person at a wedding using Microsoft Copilot:
SemiAnalysis President @fabknowledge went to a wedding this weekend in Fort Worth, Texas.
He met someone who claimed to be AI-pilled, so of course he wanted to talk to Doug about AI.
He told Doug the technology is crazy, and he couldn't believe the rest of the world wasn't talking about it.
When Doug asked what he used, he said Microsoft Copilot.
BREAKING NEWS🚨: NVIDIA just submitted its first Rubin NVL72 Kimi K3 submission, and it can reach over 850 tok/s/user, even at 2.8 Trillion parameters, with over 3x higher throughput on @vllm_project!
Great work by the NVIDIA team, and excited for Rubin Software 1.0 GA coming soon!
Alibaba's Panjiu 128 is not a true rack scale up. Panjiu 128 is made up of two 64 scale up size systems with ALink switch cartridges in the back. Alibaba's own architecture write-up says a single 128 domain would need 128-port switch chips, and the ICN Switch 1.0 is a 25.6 Tbps part sized for 64. We covered this during our Alibaba Apsara Conference takeaways note where we found that only Panjiu 144, with switches in the middle of the rack, is a true rack-scale domain which is not due until 2028. (1/2)🧵
Congrats to the SGLang and RadixArk team for their amazing work on MegaKernel + PD, improving both high concurrency throughput and low-latency serving as well! (1/2)🧵
