#computer

Over the last week I built a bottoms up SpaceX financial model through the end of 2028 The numbers that came out are insane There are wide error bars given how quickly things are changing but I wanted to distill something complicated into something that can at least be debated * This is not investment advice!!! ** Sequoia is a major SpaceX shareholder *** Past performance is not indicative of future results

Intelligence is never born in a vacuum; it is forged at the precise boundary where numbers meet machinery. The fundamental equation, AI = Data + Algorithms + Compute, captures how raw inputs and logic transform into functional reasoning. Through this elegant triad, isolated bytes evolve into adaptive systems capable of autonomous thought. Computation is simply the heartbeat of modern logic.

Reverse an audio recording, and its Fourier magnitude spectrum remains exactly the same. For a real-valued recording, reversing the samples changes the phase of the Fourier coefficients, but preserves their magnitudes. This is a fascinating property of Fourier analysis, and it reveals an important problem in signal processing. A magnitude spectrum tells us how strongly different frequencies are represented in a recording. But it cannot uniquely tell us when those components occurred. The complete complex Fourier transform retains enough information to reconstruct the original samples. The magnitude spectrum alone does not. This is where spectrograms become useful. A spectrogram is commonly computed as the squared magnitude of the Short-Time Fourier Transform (STFT), allowing engineers to examine how a signal's frequency content changes over time. Instead of analyzing an entire recording as one sequence, engineers divide the signal into short, usually overlapping windows and compute a Fourier transform for each section. But there is an important mathematical tradeoff. Short windows provide better time localization, making it easier to identify brief events. However, they provide poorer frequency resolution. Longer windows improve the ability to distinguish nearby frequencies, but make it harder to determine precisely when a change occurred. This is the fundamental time-frequency resolution tradeoff. It matters in real engineering applications. In speech processing, spectrograms help reveal consonant transitions and the formant structure of vowels. In mechanical engineering, they help reveal transient vibrations, changing rotational frequencies, and patterns that may be obscured in an average frequency spectrum. In radar and sonar, time-frequency analysis helps engineers study signals whose frequencies change over time. Even increasing the number of FFT points through zero-padding cannot recover the ability to resolve closely spaced frequencies lost by using a short observation window. The mathematics places a real constraint on what we can measure. A spectrogram is therefore more than a picture of frequencies changing over time. The choice of analysis window determines which events and frequency differences we can resolve. And that choice is an engineering decision, not simply a visualization setting.

Access latency across the memory hierarchy 😎👇 Find high-res pdf books with all my technology related infographics from #tech #technology #computer #computerscience #hardware

Our TPUs just phoned home... from space. ICYMI, on October 1, we sent four TPUs into orbit to test whether our AI hardware could operate in space. The initial goal was to see if our chips could turn on, process a prompt, and transmit a response back to Earth. This week, we got our answer. ✨ This milestone brings us one step closer to understanding if space-based compute is possible. We’re sending two more TPU-equipped satellites to orbit in 2027 for our next milestone. Watch this space 👀 and learn more about the first milestone of our long-term moonshot with Project Suncatcher ↓

Nvidia RTX 20, 30 and 40-series GPUs can now unlock their full AI performance thanks to a new UEFI-level tool. The open-source project, called nv-uefi-unlocker, works before Nvidia’s drivers load and removes a hardware-level performance limit affecting certain Tensor Core operations. The reported gains are impressive: >RTX 4090: up to 2x peak AI compute performance. >RTX 3090 Ti: up to 80% higher matrix performance. >RTX 4060 Laptop: around 65% improvement in reported tests. It supports Windows and Linux and doesn’t require a modified Nvidia driver or a permanent VBIOS flash. Sadly these are AI compute improvements, not gaming FPS gains

"The Surprising Effectiveness of Shared Memory in Looped Transformers" Looped Transformers usually maintain separate KV-caches for each pass, but this paper shows that sharing memory across loops can actually improve model quality while reducing memory usage. So Looped Prediction Transformer (LPT) trains the first loop to build a shared KV-cache, while later loops reuse it and keep only a small window of their own cache to refine predictions. Across 150M-1B models, LPT improves both perplexity and accuracy while using up to 79% less context memory in its hybrid variant. At 1B, the hybrid achieves 10% lower perplexity than a standard Transformer with five loops, though it uses a lot more inference compute.

Inference Engineering is becoming one of the most important skill sets in AI, and I think a lot of engineers are still underestimating it. Training gets most of the attention because that is where the model is created. But once a model has to serve real traffic, a different class of problems appears: queueing, batching, KV-cache pressure, GPU memory, scheduling, communication and tail latency. Inference is not one homogeneous workload. Prefill processes the prompt and builds the KV cache. Work across prompt positions can be parallelized efficiently, so prefill usually has higher arithmetic intensity and can use GPU compute. Decode is different. Each sequence advances autoregressively, usually one token per step, while repeatedly reading model weights and accumulated KV state. At practical serving batch sizes, decode can become heavily constrained by memory bandwidth. That split explains many inference optimizations. FlashAttention reduces memory traffic inside attention. PagedAttention manages KV-cache memory in blocks, reducing fragmentation. MQA and GQA reduce the number of key/value heads, cutting KV-cache storage and bandwidth. Continuous batching lets requests enter and leave the active batch as generation progresses. Chunked prefill breaks long prefills into smaller pieces so they can run alongside decode work. Prefix caching reuses KV state for repeated prefixes, saving prefill computation, but not making later decode tokens faster. Memory is another constraint. Model weights are only part of the footprint. For conventional attention, KV-cache size grows with active tokens and depends on layers, KV heads, head dimension and bytes per element. Quantization can reduce weight memory and bandwidth, while KV-cache quantization can reduce cache memory and traffic. Speculative decoding proposes tokens using a cheaper draft mechanism and verifies them with the target model. Speedup depends on draft cost, acceptance rate and verification efficiency. At larger scale, tensor parallelism adds communication, pipeline parallelism adds scheduling and bubbles, and MoE serving adds routing, communication and load-balancing problems. Some systems separate prefill and decode onto different GPU pools, paying the cost of KV transfer between them. This is why tokens per second is not enough. Production systems care about TTFT, ITL/TPOT, throughput, queueing delay, concurrency, KV-cache occupancy, GPU utilization, tail latency and goodput: how much request load still meets the latency objective. A server can show excellent aggregate throughput while users are still waiting too long for the first token. Inference engineering sits between ML systems, distributed systems, GPU architecture, networking, scheduling and performance engineering. The model decides what token should come next. Inference engineering determines how much serving that token costs, how quickly it reaches the user, and how many users the hardware can support at once.

1/ Introducing x402 Work: micropayments powered by your CPU, GPU, or ASIC. Cards, accounts, and wallets all require a human in the loop. Compute doesn't. With x402 Work, any device pays per request with its own hardware. No signups, no pre-funded wallets, no keys. Give it to your agent in an instant.

🚨 Everyone with an NVIDIA RTX card, you have to see this!!! It's causing a real uproar overseas, but it hasn't spread much yet, so I'm sharing it! → A few days ago, NVIDIA's "AI compute limit" on RTX was unlocked by a newly released open source tool → It's called GPU-Unlock → No new graphics card, no overclocking. Just lifting the restriction Gains so far: → RTX 3060: +78~99% → RTX 3070: +72~81% → RTX 3080: +68~73% → RTX 3090 Ti: BF16 +80% → RTX 4070 Ti: +83~84% → RTX 4090: specific Tensor nearly 2x → RTX 2080 Ti: +36~64% → It's not just the 3000 series. 20, 30 and 40 series, desktop and laptop, real world tests keep piling up right now If you do AI image generation or run local LLMs, you'll feel it the most Why it's blowing up: → There was locked away AI performance inside the RTX card sitting at home, and you could have been using it all along The biggest suspicion right now: → NVIDIA intentionally nerfed FP16/BF16 to create tiers between GeForce and expensive professional GPUs → NVIDIA hasn't officially acknowledged it yet Imagine learning your own card was throttled this hard. 100% open source.

Modders have unlocked hidden AI compute performance on NVIDIA’s RTX 20, 30 and 40-series GPUs. The modifications remove software-level limits on certain Tensor Core operations, without changing the hardware or flashing a new VBIOS. - RTX 4090: 165.6 → 331.6 TFLOPS FP16 Tensor Core throughput -RTX 3090 Ti: 87.5 → 157.4 TFLOPS BF16 performance - RTX 2080 Ti: 59.9 → 111 TFLOPS FP16→FP32 performance - RTX 3090: 73.0 → 122.8 TFLOPS in a PyTorch GEMM test The gains are mainly relevant to AI and compute workloads, not gaming.

switch arrived and ethernet cables arrived. it’s local compute time, gamers.

A single RTX 3090 can reportedly run Qwen3.8-27B at over 1,000 tokens/s aggregate throughput, serving 64 concurrent sessions. No multi-GPU setup. No expensive server hardware. Just one 24GB RTX 3090, capped at 250W. The project behind this is HyperQwen, an open-source inference setup built around vLLM. Its reported performance: • GPU: RTX 3090, 24GB VRAM • Power limit: 250W • Single-session generation: approximately 127 tokens/s • 64 concurrent sessions: approximately 1,035 tokens/s aggregate • Longest tested context: up to 150K tokens The distinction between those two speed figures is important. The 127 tokens/s figure represents generation speed for a single chat. The 1,035 tokens/s figure is the combined throughput across 64 simultaneous sessions. It does not mean one user gets 1,035 tokens/s. Instead, the system distributes its available compute across multiple requests, keeping the GPU busy while handling concurrent users. Why this matters Many local inference demonstrations focus on how quickly one person can generate text. HyperQwen explores a different question: how much total workload can one consumer GPU handle when several people or applications use it simultaneously? That makes the setup interesting for developers building local AI services, shared inference environments and multi-agent workflows. Getting started The project also makes deployment relatively straightforward. It provides two Docker Compose profiles: • Single-session mode: docker compose --profile single • Batch mode: docker compose --profile batch One thing to prepare for before running it: The first launch reportedly downloads approximately 9.5GB of Docker image data, followed by a local model requantization process involving roughly 20GB of data. Let that process finish before running performance tests. The accompanying demonstration shows the single-stream output first, then switches to 64 streams running concurrently. Project: HyperQwen
Source: GitHub, syv-ai/HyperQwen It’s another example of how inference software and serving optimizations can change what a 24GB GPU can do. If you’re running Qwen3.8-27B locally on a 3090, what’s your single-session generation speed?

DOS #oc #computer

"Towards Looped Models Done Right" Not all ways of looping Transformers work equally well. This paper finds that looping only the middle layers performs better than repeatedly looping the entire Transformer. The researchers compare different loop architectures and find that the Huginn-style design, which places recurrent layers between separate input and output layers, generally performs better, especially on reasoning tasks. At 8B parameters, Huginn MoE outperforms Ouro on 8 out of 10 benchmarks. It also uses 75% fewer stored parameters than a compute-matched 32B feedforward MoE, while achieving better GSM8K accuracy (83.6% vs. 80.8%)

𝗧𝗼𝗽 𝟭𝟲 𝗔𝗪𝗦 𝗦𝗲𝗿𝘃𝗶𝗰𝗲𝘀 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝘀 𝗦𝗵𝗼𝘂𝗹𝗱 𝗞𝗻𝗼𝘄. Understanding AWS isn't about memorizing service names. It's about knowing the role each one plays in your architecture. 𝗖𝗼𝗺𝗽𝘂𝘁𝗲 𝗿𝘂𝗻𝘀 𝗰𝗼𝗱𝗲: ↳ Lambda runs event-driven functions with automatic scaling. ↳ ECS orchestrates containers using AWS-native tooling. ↳ EKS provides Kubernetes with a managed control plane. 𝗗𝗮𝘁𝗮 𝘀𝘁𝗼𝗿𝗲𝘀 𝘀𝘁𝗮𝘁𝗲: ↳ RDS manages relational databases. ↳ DynamoDB provides serverless key-value and document storage. ↳ ElastiCache provides managed in-memory caching for low-latency access. 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻 𝗺𝗼𝘃𝗲𝘀 𝘄𝗼𝗿𝗸: ↳ SQS provides durable queues for asynchronous tasks. ↳ SNS handles pub/sub messaging and fanout. ↳ EventBridge routes events between applications and services. ↳ Step Functions orchestrates workflows with retries, waits, and branching. 𝗧𝗿𝗮𝗳𝗳𝗶𝗰 𝗰𝗼𝗻𝗻𝗲𝗰𝘁𝘀 𝘀𝗲𝗿𝘃𝗶𝗰𝗲𝘀: ↳ API Gateway manages APIs with authorization and throttling. ↳ CloudFront caches content closer to users. ↳ VPC provides isolated virtual networks for cloud resources. 𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆 𝗮𝗻𝗱 𝗼𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀 𝗸𝗲𝗲𝗽 𝘀𝘆𝘀𝘁𝗲𝗺𝘀 𝗰𝗼𝗻𝘁𝗿𝗼𝗹𝗹𝗲𝗱: ↳ IAM manages permissions through policies and roles. ↳ CloudWatch collects metrics, logs, and operational telemetry. ↳ Secrets Manager stores and helps rotate application credentials. Building scalable infrastructure is only part of the challenge. As software companies integrate AI into their products, they also need to connect models to existing data, manage AI workloads, and keep them secure and reliable in production. AWS supports this through services like Amazon Bedrock for building generative AI applications and Amazon Bedrock AgentCore for deploying and operating AI agents at scale. AWS's AI resources help software companies explore how to integrate AI into their products, move from prototypes to production, and scale AI-powered features across customers. 𝗘𝘅𝗽𝗹𝗼𝗿𝗲 𝗔𝗜 𝗳𝗼𝗿 𝘀𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗰𝗼𝗺𝗽𝗮𝗻𝗶𝗲𝘀 → What else would you add? —— ♻️ Repost to help others learn AWS. 🙏 Thanks to @awsdevelopers for sponsoring this post. ➕ Follow me ( Nikki Siapno ) to improve at AI and system design.

AWS and GCP are committing absolute robbery with their GPU prices Introducing compute pools > rent GPUs across 10+ regions > H100s from 2.75/h > 8xB300s available at this very moment > Mount portable NVMe drives to any instance You don't need to raise 100 million dollars to rent GPUs anymore - we're making compute available to everyone

Why temperature 0 isn't deterministic in LLM inference? (a popular technical LLM interview question) At temperature 0, inference engines usually switch to greedy decoding directly. At every step, they pick the highest-logit token, so the same prompt should give the same output every time. In practice, it doesn't. Thinking Machines actually tested this by sending "Tell me about Richard Feynman" to Qwen3-235B 1,000 times at temperature 0 and got 80 unique completions. While picking the biggest logit is a perfectly deterministic step... ...the variation actually comes from computing the logits themselves. If those numbers come out slightly different, argmax can pick a different token. Let's understand how the logits can slightly change at times. 1) The order of addition changes the result: GPUs store numbers with limited precision. fp16 keeps only 11 significant bits, and between 1024 and 2048, that leaves no room for anything after the decimal point. So 1024.4 gets rounded back to 1024. Now consider 1024 + 0.4 + 0.4 in two orders. Left to right, 1024 + 0.4 rounds back to 1024, and adding 0.4 again rounds back to 1024. The answer is 1024. If the small numbers go first, 0.4 + 0.4 gives 0.8, and 1024 + 0.8 rounds up to 1025. The answer is 1025. An LLM performs billions of such additions in every forward pass, so if their order changes, the logits change slightly too. 2) What causes the order to change? The popular reason is that parallel GPU threads finish in an unpredictable order. But that isn't the main reason, since the kernels in a typical forward pass compute each sum in a fixed order. So the same matrix multiplication on the same inputs, at the same batch size, gives bit-for-bit identical results every time. 3) The real reason is the other requests on the server: An inference server groups requests from many users into a batch and runs them together. The kernels decide how to split their work based on the batch size. - With a large batch, every GPU core is busy, so each core computes its sum in one go. - With a small batch, many cores would sit idle, so the kernel cuts each long sum into chunks, spreads them across cores, and combines the partial results at the end. A different split leads to a different order of addition, and the batch size depends on how many other people are using the server at that moment. 4) How this tiny difference changes the entire answer: Consider two runs of the same prompt: In a batch of 8, say the logits come out as: - "Queens" = 12.3417 - "New" = 12.3415 In this case, argmax picks "Queens". In a batch of 32, the rounding works out slightly differently, giving: - "Queens" = 12.3414 - "New" = 12.3416 In this case, argmax picks "New". The output only flips at such near ties. And every token after that is generated from a different prefix. In the above test by Thinking Machines, all 1,000 runs matched for the first 102 tokens. At token 103, 992 runs wrote "Queens, New York" while 8 wrote "New York City". 5) The solution: This isn't a big problem per se, but to make it genuinely deterministic, every kernel should split its work the same way regardless of batch size (for attention, that means a fixed chunk size), even if it leaves some cores idle. Thinking Machine tested this with batch-invariant kernels, where all 1,000 runs were identical. But this impacted speed. On Qwen3-8B, default vLLM took 26s for 1,000 requests, while the deterministic version took 55s, and 42s after an improved attention kernel. The visual below depicts what we discussed. That said, batching isn't the only thing that changes how these sums get split. A cache hit does too, since some attention kernels process cached tokens separately from new ones, and Thinking Machines had to handle this case explicitly. I recently covered the four caches in LLM serving in an article, with code for each. Read it below.

📱 35B parameters. Real-time inference. On a phone. Meet Lookahead Expert Scheduling (LES), the inference system powering MiniCPM-V 4.7’s 35B-A3B MoE model on edge devices. 🚀 On an iPhone 18 Pro Max, the full system reaches 20.51 tokens per second, a 6.07× speedup over loading expert weights on demand. How? Three coordinated optimizations: - 🔮 Scheduling: Predict and prefetch the next layer’s experts while the current layer computes. - 💾 Storage: Combine contiguous expert layouts with mixed-precision quantization to reduce loading overhead. - ⚡ Compute: Use DSpark speculative decoding to verify multiple candidate tokens in one pass. Predictions guide prefetching; the model’s actual router still determines which experts execute. The architecture, expert counts, and routing remain intact. Mixed-precision quantization reduces the weight file by 26.7%, with just a 0.16-point drop in the five-benchmark average. LES shows how better inference systems can bring larger, more capable models to everyday devices. 🌍 👉 LES: 🔗 GitHub: #MiniCPM #OnDeviceAI #MoE #EdgeAI

The rate card doesn't show you everything. Moving data out of a cloud, to your own machine, to another provider, or to your users, often carries its own charge, separate from compute. It's called egress, and it's one of the easiest costs to miss when comparing providers on price alone. Before you commit, ask what it costs to leave, not just what it costs to start. See the full picture →

Shaun Maguire’s bottoms-up SpaceX $SPCX model (50th percentile) shows a clear inflection starting in 2027. I converted the quarterly figures into charts, and the 2027 revenue explosion stands out immediately. Key points: • ⁠FY2026 revenue: ~$51.5B • ⁠FY2027 revenue: ~$274B (≈5.3× year-over-year) • ⁠FY2028 revenue: ~$731B • ⁠Driver: Space Infrastructure (SI), especially terrestrial compute • ⁠Terrestrial compute capacity scales from 1 GW (early 2026) to 21 GW (end-2028) • ⁠Assumption: ~$40B annual revenue per GW • ⁠By FY2028, SI is ~85% of total revenue • ⁠Operating income turns positive in Q3’26, accelerates from 2027 • ⁠EBITDA reaches ~$536B in FY2028 • ⁠Free cash flow stays negative through 2027, turns positive mid-2028 amid ~$369B of annual capex • ⁠Starship ramps to 250 flights in 2028 Shaun himself noted this is a 25th–50th percentile estimate that factors in risks like political headwinds and major supply chain issues. Elon replied: “I think we can do better.”

@SPCX100T Terrestrial compute will be peanuts compared to orbital compute. Orbital compute deployed per month probably exceeds terrestrial around the end of 2028 and then far exceeds it in 2029.