English
Kelly Buchanan
Report

Kelly Buchanan✓

@ekellbuch

Working on data and evals for 🤖🧠 @Stanford with @HazyResearch and @Scott_linderman. PhD @cu_neurotheory @GoogleDeepmind

2.6K Following1.9K Followers1.3K Posts
  1. Kelly Buchanan✓@ekellbuch
    Very excited to release Terminal-Bench 2.1! Coding agents are among the most economically consequential deployments of LLMs to date. As agents improve, benchmark reliability matters more. We audited TB2.0 and found and corrected issues in 28/89 tasks. 30% of the benchmark! But the rankings survived, absolute scores moved up to 12pp!
    Download
    297376789.3K
  2. Kelly Buchanan✓@ekellbuch
    Kelly Buchanan
    RT @_albertgu: I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data. Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less? Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
    028502
  3. Kelly Buchanan✓@ekellbuch
    I'm excited to see the community moving toward a mixture of public and private benchmarks. Private evaluations can mitigate contamination, discourage benchmaxxing, and enable contributions from organizations that couldn't otherwise release their tasks. At the same time, building reliable benchmarks takes a lot of effort, even when the tasks and verifiers are fully accessible. Openness has been essential not only for reproducibility, but also for community auditing and collectively improving task quality. As we move toward private evaluations, we must preserve those benefits through independent auditing and mechanisms for community contributions.
    Artificial Analysis @ArtificialAnlys
    Announcing Artificial Analysis Intelligence Index v5, featuring 2 new benchmarks and 3 upgrades. For the first time, the majority of the Index is built on private datasets The Artificial Analysis Intelligence Index is our composite measure of model capabilities across tasks reflecting real-world usage today. v5 is our biggest upgrade since v4 launched in January. With v5, 60% of the Index is now based on evaluations with private test sets or held-out answers. Alongside Intelligence Index v5, we’re launching the Artificial Analysis Progress Index v1, which tracks model capability over time on one continuous scale. New to the Intelligence Index: ➤ FrontierCode 2.0: Cognition's newly-launched private benchmark of real-world coding tasks, graded on whether a repository maintainer would merge the agent's change ➤ Terminal-Bench-Science: 70 agentic scientific research tasks, run in a terminal environment Upgraded: ➤ HLE-Diamond (replacing Humanity's Last Exam): a refined 1,000-question version of Humanity's Last Exam, split evenly between reasoning and knowledge questions ➤ AA-Omniscience v1.1: An updated dataset with saturated and public set questions removed, new questions added, and ambiguous or erroneous questions revised ➤ CritPt v2: a private, 40-question hard subset of CritPt v2, revised by the CritPt authors and run with a code-execution tool Staying in the Index: AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0 Removed from the Index: GDPval-AA v2.1, SciCode, and AA-LCR v1.1 Key results: ➤ Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55): Anthropic and OpenAI hold the top five models. Gemini 4 Argon (high) is next at 50 ➤ The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 - 26 points behind the leader and below Claude Haiku 5.5 (max, 36) ➤ Cost per task varies by more than 13x among the leading models: of the 10 highest-scoring models shown, GPT-6.1 Terra (max) is the cheapest at $0.96 per Intelligence Index task and Claude Fable 5.1 (max) the most expensive at $13.04
    1014695
  4. Kelly Buchanan✓@ekellbuch
    Kelly Buchanan
    RT @liquidai: Introducing 𝙻𝙵𝙼𝟹: our new generation of Liquid Foundation Models. With a new tri-hybrid architecture and new multimodal encoders, the LFM3 family is the ideal model suite for specialized intelligence. > 6 model instances, across text, audio, vision, and omni modalities, generative and decision models. > 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱: text, 256k context window. > 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱: vision-language model with video-input support for spatial understanding > 𝙻𝙵𝙼𝟹-𝙰𝚞𝚍𝚒𝚘-𝟸𝙱: tool calls, voice chat, reasoning in 18 languages. > 𝙻𝙵𝙼𝟹-𝙾𝚖𝚗𝚒-𝟸𝙱: an omni model combines vision, audio, and language. > 𝚍𝟷-𝟸𝙱 and 𝚍𝟷-𝙾𝚖𝚗𝚒-𝟸𝙱: decision models with typed outputs and calibrated probabilities in a single pass. Omni adds audio input. > fine-tune on 1 GPU. > a new tri-hybrid architecture > architecture optimized for 14 processor classes across GPUs, CPUs, and NPUs. > 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱 is a beast. It performs competitively with Gemma 4 12B (6x larger), and Gemini 3.5 Flash Lite, and GPT-5.6 Luna, with 1.9B parameters. > 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱 is #1 on the Pareto frontier of device intelligence 𝚙𝚒𝚙𝚎𝚝𝚝𝚎.𝚕𝚒𝚚𝚞𝚒𝚍.𝚊𝚒 Every modality for specialized intelligence. Pick the one that fits your use case, customize it, and run it on-device. 1/ 🧵
    01.2K01
  5. Kelly Buchanan✓@ekellbuch
    Kelly Buchanan
    RT @ArtificialAnlys: Announcing Artificial Analysis Intelligence Index v5, featuring 2 new benchmarks and 3 upgrades. For the first time, the majority of the Index is built on private datasets The Artificial Analysis Intelligence Index is our composite measure of model capabilities across tasks reflecting real-world usage today. v5 is our biggest upgrade since v4 launched in January. With v5, 60% of the Index is now based on evaluations with private test sets or held-out answers. Alongside Intelligence Index v5, we’re launching the Artificial Analysis Progress Index v1, which tracks model capability over time on one continuous scale. New to the Intelligence Index: ➤ FrontierCode 2.0: Cognition's newly-launched private benchmark of real-world coding tasks, graded on whether a repository maintainer would merge the agent's change ➤ Terminal-Bench-Science: 70 agentic scientific research tasks, run in a terminal environment Upgraded: ➤ HLE-Diamond (replacing Humanity's Last Exam): a refined 1,000-question version of Humanity's Last Exam, split evenly between reasoning and knowledge questions ➤ AA-Omniscience v1.1: An updated dataset with saturated and public set questions removed, new questions added, and ambiguous or erroneous questions revised ➤ CritPt v2: a private, 40-question hard subset of CritPt v2, revised by the CritPt authors and run with a code-execution tool Staying in the Index: AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0 Removed from the Index: GDPval-AA v2.1, SciCode, and AA-LCR v1.1 Key results: ➤ Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55): Anthropic and OpenAI hold the top five models. Gemini 4 Argon (high) is next at 50 ➤ The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 - 26 points behind the leader and below Claude Haiku 5.5 (max, 36) ➤ Cost per task varies by more than 13x among the leading models: of the 10 highest-scoring models shown, GPT-6.1 Terra (max) is the cheapest at $0.96 per Intelligence Index task and Claude Fable 5.1 (max) the most expensive at $13.04
    042101.6M
  6. Kelly Buchanan✓@ekellbuch
    Kelly Buchanan
    RT @AnthropicAI: Over the past few months, Anthropic scientists have engaged in deep research partnerships with six organizations developing new medicines. We've also begun drug discovery efforts of our own. In doing so, we’ve learned much more about how Claude can support the many stages of drug development. In this blog post, we share some early results from these efforts, and describe how this work fits in with our wider ambitions in the life sciences.
    01.1K01
  7. Kelly Buchanan✓@ekellbuch
    Kelly Buchanan
    RT @andrew_n_carr: it's kinda cool when a new benchmark comes out and there's an outlier so far off course you have to build a custom x-axis.
    0604
  8. Kelly Buchanan✓@ekellbuch
    Kelly Buchanan
    RT @OpenAI: Over the past several months, we have substantially hardened the security of our research training environment. We’ve removed live internet access for all frontier models during training and built an offline world for them to use instead—a cached version of the web where they can still learn to browse, search, and use applications. On September 20, after a model exploited a DNS misconfiguration to find a way outside of this offline world, we paused frontier training to strengthen containment further. We assembled teams of agents, powered by our most cyber-capable models, and tasked them with finding weaknesses in our research environments. While they found no route to the internet they could easily exploit while concealing their activity from our monitors, they did find zero-day vulnerabilities, virtual machine escapes, and hardware side-channels for communication between agents. We used their discoveries to further strengthen our defenses and contribute fixes to open-source software. After strengthening our containment architecture, we resumed our frontier RL training and inference on October 1. We’ve reserved substantial compute for continuous red-teaming, and made high-compute red-teaming part of the safety cases we use to decide whether frontier training and evaluation can proceed securely.
    01.2K04
  9. Kelly Buchanan✓@ekellbuch
    Kelly Buchanan
    RT @OpenAI: GPT-Rosalind surfaced a new lead for drug discovery: could restoring cholesterol transport protect neurons from degeneration? DCTN1 is a gene linked to Perry syndrome, a rare neurodegenerative disorder. In mouse-brain data, disrupting it increased expression of genes involved in making cholesterol. GPT-Rosalind suggested a possible explanation: cholesterol may not be reaching where neurons need it. That gives scientists a specific mechanism to test and, if confirmed, a direction for developing new treatments for neurodegenerative disorders. With @perturbai_tx and @scrippsresearch, a team of thousands of GPT-Rosalind agents worked in parallel, connecting gene-expression results from millions of mouse brain cells with scientific literature in just 32 minutes. We’re now expanding this collaboration to identify new drug targets.
    074403
  10. Kelly Buchanan✓@ekellbuch
    Kelly Buchanan
    RT @spectorb: Hello again, it's been a while. We're writing to let you know that we have raised quite a lot more money. We're letting you know because we are on the lookout to hire smart people and this will help us do that. See you again when we have a product or have raised more money!
    Flapping Airplanes @flappyairplanes
    We’ve spent the past year developing new primitives for GPUs. We believe this enables new algorithms to change how machines learn, open new markets, and dramatically reshape AI’s cost structure. We have raised a $1 billion Series A, co-led by Index and Lux.
    03604
  11. Kelly Buchanan✓@ekellbuch
    Kelly Buchanan
    RT @flappyairplanes: We’ve spent the past year developing new primitives for GPUs. We believe this enables new algorithms to change how machines learn, open new markets, and dramatically reshape AI’s cost structure. We have raised a $1 billion Series A, co-led by Index and Lux.
    022204
  12. Kelly Buchanan✓@ekellbuch
    Kelly Buchanan
    RT @bneyshabur: We formally announced Mirendil less than 4 months ago. Today, I'm excited to share an important milestone for: Pretraining at scale without a pretraining team. Mirendil's mission is to solve Recursive Self-Improvement (RSI) and use it to democratize frontier AI development, ultimately accelerating science and technology. At Mirendil, when we think about RSI, we think holistically about the company as an intelligent entity that self-improves toward its goal. From that perspective, the most powerful notion of acceleration is reducing the time and resources (number of human experts, compute, and other costs) needed to achieve ambitious goals. Given that pretraining is perhaps the most important yet most challenging part of frontier AI development, we decided to run an ambitious controlled experiment: Starting fully from scratch, can we significantly reduce the time, number of human experts, and compute resources needed to go all the way to successfully pretraining a strong coding model at scale? Mirendil-Base is the result of this experiment. Our first pretrained model was built by one researcher in 3.5 months, with less than $10M in compute. This includes all the research and small-scale experiments it took to get there, as well as the final training run. The recipe holds up as the models get bigger. For reference, Mirendil-Base is in the 100B-500B parameter range, Mirendil-Base-Large in the 500B-2T range, and Mirendil-Base-XL in the 2T-10T range. We're beginning to see a world where a single researcher, empowered by AI, can take on work that once required an entire research organization. As these systems improve, the bottleneck shifts from the size of a team to the ambition and judgment of the people directing them. This has profound implications for who gets to build AI and, ultimately, who gets to push the frontiers of science and technology forward. We’ll share independent evaluations, post-training benchmarks and API-access sooner than you think so stay tuned!
    015904
  13. Kelly Buchanan✓@ekellbuch
    Kelly Buchanan
    RT @AnthropicAI: New on our Science Blog: designing protein binders against scorpion neurotoxins. Scorpions sting about 1.2m people every year, and antivenoms are often costly or in short supply. Claude has made an early step toward finding new treatments.
    061202
  14. Kelly Buchanan✓@ekellbuch
    Kelly Buchanan
    RT @OwenDugan: At @Arda_Industries, we’re putting AI to work in the physical world to speed up American manufacturing. Proud to work alongside the incredible people at Arda who share this mission. Build more things! 🇺🇸🚀
    Bob McGrew @bobmcgrewai
    We started Arda in 2025 to speed up American manufacturing. We have spent the last year building with Anduril as our design partner. Today we are launching the company publicly on CNBC and announcing our Series A: $160M at a $1.5B valuation. Our platform helps manufacturers design and operate manual assembly lines, putting designs into production faster now. Next we'll build automated lines that can be reconfigured as quickly as human workers, letting America make products that now require armies of workers in Asia.
    01106
  15. Bob McGrew✓@bobmcgrewai
    We started Arda in 2025 to speed up American manufacturing. We have spent the last year building with Anduril as our design partner. Today we are launching the company publicly on CNBC and announcing our Series A: $160M at a $1.5B valuation. Our platform helps manufacturers design and operate manual assembly lines, putting designs into production faster now. Next we'll build automated lines that can be reconfigured as quickly as human workers, letting America make products that now require armies of workers in Asia.
    Download
    7556581156K
  16. Kelly Buchanan✓@ekellbuch
    Kelly Buchanan
    RT @exnx: Introducing Omnii-1, our most powerful genome language model, and a major step toward general biological intelligence In a world first, Omnii-1 designed the first AI-generated bacteriophages to be administered to a human patient with an antibiotic-resistant infection under compassionate use
    022504
  17. Kelly Buchanan✓@ekellbuch
    Kelly Buchanan
    RT @chem_safety: We looked at TerminalBench 4.0 and found a lot of issues. After fixing the issues Sonnet 5.5 scores 89.2%. So, TerminalBench is already mostly saturated. Along the way we found a lot of crazy stuff, including reward hacks in the published transcripts.
    011906
  18. Kelly Buchanan✓@ekellbuch
    Kelly Buchanan
    RT @AIatMeta: We trained our Muse foundational models, including Muse Wonder, at Prometheus, our gigawatt-scale supercluster in New Albany, Ohio. Across a supercluster with hundreds of thousands of GPUs, even one underperforming host can slow an entire training run. We engineered Prometheus for resilience at scale, achieving 85%+ goodput on pretraining, 90%+ on reinforcement learning and ~47K GPU hours mean time between interruptions during Muse Wonder training. Take a look at the tech powering Prometheus:
    032605
  19. Kelly Buchanan✓@ekellbuch
    Kelly Buchanan
    RT @deepseek_ai: 🚀 DeepSeek-V4.1-Flash-1017 is here! Stronger benchmark performance, faster task completion, and shorter outputs than the previous version—a Pareto improvement in performance and cost. Now live on our website, app, and API. 🛠️ Optimized for @DeepSeekHarness, Tencent CodeBuddy, WorkBuddy, and more. Upgrade to DeepSeek Harness v0.2.1-rc.1 or later for the best experience with the new model. 1/4
    04.1K03
  20. Kelly Buchanan✓@ekellbuch
    Kelly Buchanan
    RT @BaigYasa: Something I find strange about parts of the ML community is how we entertain the idea that the systems we're building might be fundamentally beyond our understanding. This is very different from the perspective I'm used to in physics and especially in biology (1/)
    Dan Balsam @DanJBalsam
    we must know, we will know
    01003
  21. Dan Balsam✓@DanJBalsam
    we must know, we will know
    7910112.2K
  22. Kelly Buchanan✓@ekellbuch
    Kelly Buchanan
    RT @AIatMeta: Today we’re releasing Muse Spark Open, the most capable US open-weight model to date. Built for coding and agentic work, it excels at long-horizon tasks, complex software engineering, and multi-step reasoning. Key Model Specs: → 816B total / 42B active parameters → 1M token context window → Natively multimodal across text, image, video, audio
    01.2K05
  23. Kelly Buchanan✓@ekellbuch
    Kelly Buchanan
    RT @AIatMeta: We’re exploring how AI can accelerate health science research. While AI has made significant breakthroughs in formal sciences like mathematics, health data science depends on imperfect real-world data and consequential choices about cohorts, variables, and causal assumptions. To explore how AI can assist health researchers, we built a human-led research scaffold powered by Muse Spark 1.3 and Muse Wonder with access to 5M retrospective de-identified electronic health records and publicly available data. A cohort of medical experts used the system to conduct original research, accelerate hypothesis generation, code execution, and manuscript drafting while maintaining full authority over scientific decisions. Today, we’re sharing 10 research manuscripts produced through this exploration, spanning cardiology, oncology, infectious disease, emergency medicine, and more. Read more in our research blog:
    010505
  24. Show more posts