Tim Messerschmidt✓

@SeraAndroid

Agents & Developer Experience Ecosystems Lead EMEA at Google. O'Reilly author. Local AI, fine-tuning. Building tool-eval-bench & awesome-local-ai. Views my own.

1.6K Following7.2K Followers26.6K Posts
  1. Tim Messerschmidt✓@SeraAndroid
    I couldn't keep track of all the local AI tools flying around anymore. New engines, harnesses, tooling, benchmarks... every week. So I started awesome-local-ai: a list sorted by what you want to do, not by hype. The twist: it's built for agents too. Point your coding agent at it and ask it to recommend a setup for your hardware. It's early and DGX Spark-heavy. Help me fill the gaps:
    19141158.5K
  2. Tim Messerschmidt✓@SeraAndroid
    Pascal: "I have made this longer than usual because I have not had time to make it shorter." AI gave us the time. We spend it generating more. Add to AGENTS․md: 1. Answer first. 2. Cut every sentence that doesn't change what the reader knows or does. 3. No preamble, no recap.
    Download
    100163
  3. Tim Messerschmidt✓@SeraAndroid
    This is weirdly enjoyable. I’ll make my French colleagues go through a similar routine going forward 😆
    best of léa seydoux @badpostseydoux
    Léa Seydoux tries some of the finest French and British delicacies for ELLE UK.
    000176
  4. best of léa seydoux@badpostseydoux
    Léa Seydoux tries some of the finest French and British delicacies for ELLE UK.
    Download
    5562.6K60K15.3M
  5. Tim Messerschmidt✓@SeraAndroid
    Welcome, Le Chonk! Glad to see a European model competing with the open weights frontier. 10 style points for the name and graphics 🔥
    Mistral AI @MistralAI
    Today, we release the weights of a new and improved Le Chonk, advancing the open frontier for agentic coding and cybersecurity. 💻 Built to build: Le Chonk ranks #1 on @cognition’s FrontierCode among open models, and sets a new benchmark for Pareto efficiency on @artificialanalysis. 🛡 Built to defend: State-of-the-art in cyberdefense, put through the paces by the world's leading cyber vendors, including @CrowdStrike, @cloudflare, @rubrik, @zscaler, and others. Built for sovereignty: Make it yours. Download the weights on @huggingface , customize it with Mistral Forge, and deploy it wherever you need it. Read more from our team and customers 👇:
    003243
  6. Mistral AI✓@MistralAI
    Today, we release the weights of a new and improved Le Chonk, advancing the open frontier for agentic coding and cybersecurity. 💻 Built to build: Le Chonk ranks #1 on @cognition’s FrontierCode among open models, and sets a new benchmark for Pareto efficiency on @artificialanalysis. 🛡 Built to defend: State-of-the-art in cyberdefense, put through the paces by the world's leading cyber vendors, including @CrowdStrike, @cloudflare, @rubrik, @zscaler, and others. Built for sovereignty: Make it yours. Download the weights on @huggingface , customize it with Mistral Forge, and deploy it wherever you need it. Read more from our team and customers 👇:
    Download
    3076034.7K415K
  7. Tim Messerschmidt✓@SeraAndroid
    Enjoying a couple of days off with the family in Crete. My highlight so far? A loop up to the Lasithi Plateau: 91 km, 1,609 m of elevation. My biggest climb in one go outside of my Zwift Everesting. Time off to recharge, gather new ideas and invest in wellbeing. The work is better for it when I'm back.
    Download
    001137
  8. Tim Messerschmidt✓@SeraAndroid
    Vacation season, and I'm enjoying how quickly @GenkitFramework lets me turn ideas into agentic apps. It's an open-source SDK for TypeScript, Go, Dart and Python, works with any model, and its local Developer UI shows every tool call. Agent skills and the MCP server let your coding agent run your flows and read the traces.
    225248
  9. Tim Messerschmidt✓@SeraAndroid
    Great post on measuring a quant's quality by @mhelcig and @adotdad. TL;DR: Tuning only the quantization scales of a 3-bit model on the eval text took its KLD from 0.205 to 0.048, the same as the 4-bit model's. On held-out text it was 0.149, and GSM8K was 85.6% against 90.8% for 4-bit. Worth reading before you trust a quant ranking. KLD is a diagnostic, not a leaderboard. Your own tasks decide.
    Michael Helcig @mhelcig
    116465
  10. Tim Messerschmidt✓@SeraAndroid
    Introducing a new little project of mine: LabTunnel. A macOS menu bar app that forwards a port from an SSH host to localhost on your Mac. Flip a switch, open the page. Handy for MVPs and for login flows on headless devices. Agent-friendly by default.
    Download
    609555
  11. Tim Messerschmidt✓@SeraAndroid
    Posts @SeraAndroid
    For scripts and agents, there's a labtunnel CLI with JSON output and fixed exit codes, plus a bundled SKILL.md: labtunnel add --name Grafana --host homelab --remote-port 3000 --connect
    00083
  12. Tim Messerschmidt✓@SeraAndroid
    PyTorch 2.15 makes 4-bit NVFP4 decoding on Blackwell cheaper and therefore faster by removing one kernel launch per layer: the dequantization scale now folds into the GEMM. Why NVFP4 is not just another 4-bit format: 16 values share one FP8 scale (MXFP4 uses 32 values and a power-of-two scale), plus one FP32 scale per tensor. Finer scales mean less quantization error.
    PyTorch @PyTorch
    PyTorch 2.15 is here, with 3,117 commits from 583 contributors and updates across distributed training, NVGEMM, Apple Silicon, and accelerator platform support. The release makes the rewritten NCCL backend the default and adds one-sided transports with a NIXL backend. FSDP2 and pipelining gain finer memory and communication control. It also brings faster 4-bit (NVFP4) LLM decoding on NVIDIA Blackwell GPUs, FP8 and faster decode on Apple Silicon, and broader platform support. On Thursday, November 5, at 11 AM PST, join Tristan Rice (@rice_fry), Natalia Gimelshein, Huy Do, and Jane Xu (@janeyx99) (@Meta) for a live PyTorch 2.15 Q&A, moderated by Chris Gottbrath (Gottbrath Tech). The speakers will share an overview of the release and answer community questions about PyTorch and the new capabilities in this release. 🔗 Read the release blog and register for the live Q&A:
    1112784
  13. PyTorch✓@PyTorch
    PyTorch 2.15 is here, with 3,117 commits from 583 contributors and updates across distributed training, NVGEMM, Apple Silicon, and accelerator platform support. The release makes the rewritten NCCL backend the default and adds one-sided transports with a NIXL backend. FSDP2 and pipelining gain finer memory and communication control. It also brings faster 4-bit (NVFP4) LLM decoding on NVIDIA Blackwell GPUs, FP8 and faster decode on Apple Silicon, and broader platform support. On Thursday, November 5, at 11 AM PST, join Tristan Rice (@rice_fry), Natalia Gimelshein, Huy Do, and Jane Xu (@janeyx99) (@Meta) for a live PyTorch 2.15 Q&A, moderated by Chris Gottbrath (Gottbrath Tech). The speakers will share an overview of the release and answer community questions about PyTorch and the new capabilities in this release. 🔗 Read the release blog and register for the live Q&A:
    Download
    112116716.9K
  14. Tim Messerschmidt✓@SeraAndroid
    I'm pretty excited about Android MCP. Android is leaning into becoming the agent-friendly OS!
    Mishaal Rahman @MishaalRahman
    ✨ This is huge 👀: We're launching Android MCP, powered by the AppFunctions API! This lets your app integrate with @GeminiApp on Android so your users can ask questions that pull info from your app or take actions in it. To support this, we're also launching a preview of the Connected Apps Platform. This is the announcement I'm most excited about from #AndroidDevSummit!
    206448
  15. Mishaal Rahman✓@MishaalRahman
    ✨ This is huge 👀: We're launching Android MCP, powered by the AppFunctions API! This lets your app integrate with @GeminiApp on Android so your users can ask questions that pull info from your app or take actions in it. To support this, we're also launching a preview of the Connected Apps Platform. This is the announcement I'm most excited about from #AndroidDevSummit!
    Download
    451121.1K57.2K
  16. Tim Messerschmidt✓@SeraAndroid
    Yeah. No. I was very excited for the 512GB VRAM one but that's a tad too much given it would have co-existed with my four DGX Sparks.
    Download
    100202.1K
  17. Tim Messerschmidt✓@SeraAndroid
    Welcome, Gemini 4 Carbon! Massive improvements over Argon -- just two weeks after Argon was released. Give it a try. I'm sure you'll love it! 🚀
    Logan Kilpatrick @OfficialLoganK
    Introducing Gemini 4 Carbon, it's a big jump over Argon which we released just 2 weeks ago. Gemini 4 Carbon is SOTA on cyber, long horizon tasks, ranks #1 on Artificial Analysis V5, significantly improves on usability in domains like coding, and is more aligned.
    104302
  18. Logan Kilpatrick✓@OfficialLoganK
    Introducing Gemini 4 Carbon, it's a big jump over Argon which we released just 2 weeks ago. Gemini 4 Carbon is SOTA on cyber, long horizon tasks, ranks #1 on Artificial Analysis V5, significantly improves on usability in domains like coding, and is more aligned.
    Download
    1.2K1.1K15.6K1.2M
  19. Tim Messerschmidt✓@SeraAndroid
    That was a technical KO as of the second sentence of Mitchell’s response.
    Download
    Mitchell Hashimoto @mitchellh
    I like to look at claims like this in case I can improve Ghostty. I grabbed your source and immediately spotted the issue: your read loop overwrites a 64 KiB ring. Everything older is discarded unparsed per tick. You're faster because you're not actually terminal emulating. It means the number isn't a throughput number. In a kitten benchmark run at the rate you're claiming, roughly 6.6 MB arrives per 60 Hz tick and 64 KiB of it gets parsed. You're read(2)-ing 100% of the data and parsing about 1% of it. There are other minor issues (a small fixed scrollback in a preallocated ring buffer, no grapheme clustering, throwing away data like OSC 8 URIs, etc.), but the major one is simply that you're discarding ~99% of the input. For plain text, e.g. your cat benchmark, the final screen ends up looking right, but the scrollback is missing 99% of the file and each retained chunk starts mid-line. As soon as there are escape sequences in the dropped spans, your emulator is fully out of sync and invalid: a cut inside a CSI prints the parameters as text, and any SGR, mode, title, or alt-screen change in the gap is just... gone. FYI, this is why I don't like when people point at refterm as a terminal benchmark. refterm implements a TINY subset of what a terminal emulator must support, it's closer to a fast file reader, not a terminal emulator. I also don't mean any shade by this. I like to look at technology objectively to try to learn something and improve my own craft and work. So I'm sharing what I found. I modified your code very slightly to parse on every full buffer (rather than per tick). And the results are attached. Both use main commit in each repo at the time of this post. Note the graph is in log scale, the discrepancy between performance for things like Kitty images is measured in GB/s.
    001259
  20. Mitchell Hashimoto✓@mitchellh
    Posts @valvesxyz
    I like to look at claims like this in case I can improve Ghostty. I grabbed your source and immediately spotted the issue: your read loop overwrites a 64 KiB ring. Everything older is discarded unparsed per tick. You're faster because you're not actually terminal emulating. It means the number isn't a throughput number. In a kitten benchmark run at the rate you're claiming, roughly 6.6 MB arrives per 60 Hz tick and 64 KiB of it gets parsed. You're read(2)-ing 100% of the data and parsing about 1% of it. There are other minor issues (a small fixed scrollback in a preallocated ring buffer, no grapheme clustering, throwing away data like OSC 8 URIs, etc.), but the major one is simply that you're discarding ~99% of the input. For plain text, e.g. your cat benchmark, the final screen ends up looking right, but the scrollback is missing 99% of the file and each retained chunk starts mid-line. As soon as there are escape sequences in the dropped spans, your emulator is fully out of sync and invalid: a cut inside a CSI prints the parameters as text, and any SGR, mode, title, or alt-screen change in the gap is just... gone. FYI, this is why I don't like when people point at refterm as a terminal benchmark. refterm implements a TINY subset of what a terminal emulator must support, it's closer to a fast file reader, not a terminal emulator. I also don't mean any shade by this. I like to look at technology objectively to try to learn something and improve my own craft and work. So I'm sharing what I found. I modified your code very slightly to parse on every full buffer (rather than per tick). And the results are attached. Both use main commit in each repo at the time of this post. Note the graph is in log scale, the discrepancy between performance for things like Kitty images is measured in GB/s.
    Download
    1772408.3K832K
  21. Tim Messerschmidt✓@SeraAndroid
    I look forward to seeing what @MiaAI_lab and @ashxhart will do with this. Still counts as local AI, doesn’t it?
    Wall St Engine @wallstengine
    A 300-rack AI supercomputer powered by 9,600 Nvidia A100 GPUs is listed on eBay for $59.7M. That’s 1,200 GPU compute servers with 384TB of combined GPU memory and roughly 1.4 petabytes of system RAM. Each compute server connects eight A100 GPUs through $NVDA's NVSwitch technology. More than six years after A100 launched, The seller is offering these systems refurbished for new AI training and inference deployments.
    3011906
  22. Wall St Engine✓@wallstengine
    A 300-rack AI supercomputer powered by 9,600 Nvidia A100 GPUs is listed on eBay for $59.7M. That’s 1,200 GPU compute servers with 384TB of combined GPU memory and roughly 1.4 petabytes of system RAM. Each compute server connects eight A100 GPUs through $NVDA's NVSwitch technology. More than six years after A100 launched, The seller is offering these systems refurbished for new AI training and inference deployments.
    Download
    2821582.6K495K
  23. Tim Messerschmidt✓@SeraAndroid
    Google Research gave an LLM a sleep phase, and it stays near-perfect on BABILong up to 10M tokens. How it works: while awake, new facts sit in short-term memory (context, KV cache). During sleep, the model unlocks pre-allocated low-rank MoE blocks and distills that short-term memory into them, so old weights stay untouched. Then it "dreams": random experts generate practice examples, filtered by gradient, and it trains on the best ones. The part I find interesting: no fixed knowledge cutoff, and near-zero forgetting when learning Manchu and Kalamang in stages.
    Google Research @GoogleResearch
    Today on the blog, learn about our new instance of a “sleep” phase in continual learning, which enables LLMs to continuously integrate new facts and enhance reasoning while mitigating catastrophic forgetting or static knowledge cutoffs. More →
    108699
  24. Google Research✓@GoogleResearch
    Today on the blog, learn about our new instance of a “sleep” phase in continual learning, which enables LLMs to continuously integrate new facts and enhance reasoning while mitigating catastrophic forgetting or static knowledge cutoffs. More →
    Download
    732672.5K159K
  25. Tim Messerschmidt✓@SeraAndroid
    Here’s your chance to help shape the future of Gemma. Your input is much appreciated!
    Omar Sanseviero @osanseviero
    So...what would you like to see in our future Gemma models?
    115673
  26. Omar Sanseviero✓@osanseviero
    So...what would you like to see in our future Gemma models?
    569431.2K110K
  27. Tim Messerschmidt✓@SeraAndroid
    I love it when people push the boundaries of what is possible with models. Looking forward to @mmastrac releasing his take on EmbeddingGemma2 - another great use case for @googlegemma 💎
    Matt Mastracci @mmastrac
    It's been a while since I showed any Jev stuff but I've been cooking on a way to use @GoogleDeepMind's EmbeddingGemma2 as an insanely fast real-time /v1/systemone endpoint (20ms on text, 100ms on images). It's trainable in real-time, can distill an upstream teacher model, and is multi-modal by default. Hard to believe a @googlegemma model of this size can do so well with the right harness!
    108553
  28. Matt Mastracci✓@mmastrac
    It's been a while since I showed any Jev stuff but I've been cooking on a way to use @GoogleDeepMind's EmbeddingGemma2 as an insanely fast real-time /v1/systemone endpoint (20ms on text, 100ms on images). It's trainable in real-time, can distill an upstream teacher model, and is multi-modal by default. Hard to believe a @googlegemma model of this size can do so well with the right harness!
    Download
    892042.3K184K
  29. Tim Messerschmidt✓@SeraAndroid
    For the first time, most of the Artificial Analysis Intelligence Index runs on private test sets or held-out answers: 60% of v5, per AA. 70% of the weight sits in evals where the agent uses tools. Private sets keep scores out of training data, which is why tool-eval-bench lets you bring a private tool catalog. The cost is that nobody outside can check them, so a big rerank is hard to verify. Good reason to run your own private eval on the task you care about.
    Artificial Analysis @ArtificialAnlys
    Announcing Artificial Analysis Intelligence Index v5, featuring 2 new benchmarks and 3 upgrades. For the first time, the majority of the Index is built on private datasets The Artificial Analysis Intelligence Index is our composite measure of model capabilities across tasks reflecting real-world usage today. v5 is our biggest upgrade since v4 launched in January. With v5, 60% of the Index is now based on evaluations with private test sets or held-out answers. Alongside Intelligence Index v5, we’re launching the Artificial Analysis Progress Index v1, which tracks model capability over time on one continuous scale. New to the Intelligence Index: ➤ FrontierCode 2.0: Cognition's newly-launched private benchmark of real-world coding tasks, graded on whether a repository maintainer would merge the agent's change ➤ Terminal-Bench-Science: 70 agentic scientific research tasks, run in a terminal environment Upgraded: ➤ HLE-Diamond (replacing Humanity's Last Exam): a refined 1,000-question version of Humanity's Last Exam, split evenly between reasoning and knowledge questions ➤ AA-Omniscience v1.1: An updated dataset with saturated and public set questions removed, new questions added, and ambiguous or erroneous questions revised ➤ CritPt v2: a private, 40-question hard subset of CritPt v2, revised by the CritPt authors and run with a code-execution tool Staying in the Index: AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0 Removed from the Index: GDPval-AA v2.1, SciCode, and AA-LCR v1.1 Key results: ➤ Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55): Anthropic and OpenAI hold the top five models. Gemini 4 Argon (high) is next at 50 ➤ The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 - 26 points behind the leader and below Claude Haiku 5.5 (max, 36) ➤ Cost per task varies by more than 13x among the leading models: of the 10 highest-scoring models shown, GPT-6.1 Terra (max) is the cheapest at $0.96 per Intelligence Index task and Claude Fable 5.1 (max) the most expensive at $13.04
    21161.4K
  30. Artificial Analysis✓@ArtificialAnlys
    Announcing Artificial Analysis Intelligence Index v5, featuring 2 new benchmarks and 3 upgrades. For the first time, the majority of the Index is built on private datasets The Artificial Analysis Intelligence Index is our composite measure of model capabilities across tasks reflecting real-world usage today. v5 is our biggest upgrade since v4 launched in January. With v5, 60% of the Index is now based on evaluations with private test sets or held-out answers. Alongside Intelligence Index v5, we’re launching the Artificial Analysis Progress Index v1, which tracks model capability over time on one continuous scale. New to the Intelligence Index: ➤ FrontierCode 2.0: Cognition's newly-launched private benchmark of real-world coding tasks, graded on whether a repository maintainer would merge the agent's change ➤ Terminal-Bench-Science: 70 agentic scientific research tasks, run in a terminal environment Upgraded: ➤ HLE-Diamond (replacing Humanity's Last Exam): a refined 1,000-question version of Humanity's Last Exam, split evenly between reasoning and knowledge questions ➤ AA-Omniscience v1.1: An updated dataset with saturated and public set questions removed, new questions added, and ambiguous or erroneous questions revised ➤ CritPt v2: a private, 40-question hard subset of CritPt v2, revised by the CritPt authors and run with a code-execution tool Staying in the Index: AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0 Removed from the Index: GDPval-AA v2.1, SciCode, and AA-LCR v1.1 Key results: ➤ Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55): Anthropic and OpenAI hold the top five models. Gemini 4 Argon (high) is next at 50 ➤ The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 - 26 points behind the leader and below Claude Haiku 5.5 (max, 36) ➤ Cost per task varies by more than 13x among the leading models: of the 10 highest-scoring models shown, GPT-6.1 Terra (max) is the cheapest at $0.96 per Intelligence Index task and Claude Fable 5.1 (max) the most expensive at $13.04
    Download
    5864565.4K1.9M
  31. Tim Messerschmidt✓@SeraAndroid
    Posts @SeraAndroid
    Private catalogs in tool-eval-bench's toolscale:
    001117
  32. Show more posts