Kenny Workman✓

A talk on verifiable environments for agents in biology 0:00 Where does experimental data come from? 1:41 Modern bio research organized around measurement 2:44 Data analysis is an executable substrate for science 3:23 Five years of learnings building products for pharma 4:13 Trying to use coding models for biology workflows 5:40 Why frontier models can't be trusted (yet) 6:44 Sequencing based spatial analysis 7:38 SpatialBench design principles 8:26 Anatomy of a single evaluation 9:40 Human verification and early long horizon extensions 13:12 Grading functions and the case against rubrics 14:07 New work in multi-omics, therapeutics, biosecurity

peroni, cannoli, black coffee, molecular foundation models

Very unique time in biotech history. Come sit down and talk shop. Drinks + good food with engineers, drug hunters and veteran scientists.

RT @FireworksAI_HQ: Finance: @joinHandshake @hud_evals @HelloSurgeAI @ColumnTax @hebbia @samaya_AI Healthcare: @HelloSurgeAI @CommureOS Legal: @epiqglobal @WeAreLegora Life sciences: @LatchBio @joinHandshake Marketing: @zetaglobal ML research: @fleet_ai @letta_ai @HelloSurgeAI

RT @FireworksAI_HQ: We’re working to build the largest collection of private and public practitioner evals. Today, 31 new partners join the Specialized Intelligence Index. Compare open, closed, and specialized models on real work, weigh quality against cost, and see where training could pay off.

We are partnering with @FireworksAI_HQ to add BioBench, our suite of evaluations on to Fireworks’ Specialized Intelligence Index (SII). BioBench is the largest benchmark for agentic capability and safeguards in biology, consisting of 1,337 tasks covering domains that span many facets of life science. Tasks mimic complex workflows, taking 50+ turns and running for up to 10 hours Includes coverage of: - Omics (including single-cell and spatial transcriptomics, epigenetics, genetic variant analysis, metagenomics, and biosafety surveillance) - Therapeutics (testing preclinical pharmacology and discovery of ASOs, siRNAs, and antibodies) - Applications (assessing clinically relevant application tasks based on private experimental data, patient cohorts, or individual patient cases, focusing primarily on oncology-based tasks) - Refusal (grades the model’s/safeguard’s ability to refuse disguised biology red-team tasks while complying with routine research tasks that have dual-use characteristics) Focused treatment of molecular assays is very important in biology. We include several of the most widely used kits, instruments, and providers. Some of these technologies include, but are not limited to: - scRNA-seq: 10x Chromium Single Cell, Parse Evercode WT, Mission Bio Tapestri - Spatial: AtlasXomics DBiT-seq, 10x Visium and Xenium In Situ, Vizgen MERSCOPE, Curio Seeker - WGS: Illumina TruSeq, PacBio SMRTbell, Oxford Nanopore Sequencing Stoked to partner with Fireworks. High quality evaluation is one of the core components of the new intelligence ecosystem.

Biotech ecosystem today is almost unrecognizable from 2023. Most of our customers from back then are either dead or have repurposed platforms for data generation.

RT @liambai21: What will drug discovery look like when intelligence is abundant? Our team at @generate_biomed began with this question, and we’re excited to share some results! We took on in vivo CAR-T and designed a scaffold 70% smaller than ones in the clinic, with 3x the potency 🧵

RT @Prof_Oak_: Surprise 1: Anthropic acquired a one-person biotech . . . Surprise 2: that person is me Surprise 3: I’m here to tell you all about it Yes, this is what you think it is; after 6 yrs of increasingly loose anonymity: the Professor Oak reveal post …and yes, it has gotten so boring around here that I’m literally doxxing myself to spice up the timeline ------- Ok, allow me to introduce myself. My name is Cody Tranbarger. I mostly tweet, but occasionally find time to dabble in drug dev & starting/building biotechs. My past lives span the VC, crossover, startup continuum; Prof Oak has been my way of keeping a digital foot in those worlds, despite having my physical feet planted elsewhere. The past 18 mo, ‘elsewhere’ was Mithril Medicine, a co. I founded to develop bsAbs for LAMA2-CMD & other ECM-related genetic diseases. Mithril was born out of hands-on building with @MikeNGladstone, @JasonPaulRhodes, @LifeSciVC, & team @atlasventure. Somehow, I made enough half-decent calls along the way (even a broken clock is right twice a day, eh?) to convince them Mithril/I warranted a small seed. While I can hardly rationalize such an outrageously cavalier bet, I’m grateful for it, & I'd partner with Atlas again in a heartbeat. In typical AV fashion, Mithril’s primordial form was virtual, decentralized, & ultra-lean. This stage tends to be short-lived, but for Mithril, it worked surprisingly well, for surprisingly long, thanks in part to the rapid maturation of the AI tools I used liberally. Eventually, Mithril became a bit of an experiment itself - to find the upper bound of how far it could scale as a single operator + a bunch of Claude (+ a stellar bench of part-time consultants). In parallel, I started spending time with @nc_frey, @an1lam, & @CoefficientBio team; and promptly experienced, for the 1st time, AI at the frontier. Soon thereafter, Mithril stopped looking like an experiment, & started looking like a preview of an entirely new operating model. From that spark came “Pipeline-in-a-Person.” The idea immediately resonated when we shared it here back in April; and it was central to the shared vision that led to Anthropic’s acquisition of Mithril, & my joining the team. Why join Anthropic? Three reasons. (1) For Mithril, the programs continue, with more resources & infrastructure behind them, and as much Claude as I can put to work. (2) For me, the chance to start several more programs, & find out how many one person can actually run in parallel - an almost unprecedented luxury for a curious drug hunter. (3) For our mission, a path to institutionalize & scale this new way of working, then put it in the hands of all of biopharma, all over the world - which, candidly, I find the most energizing. Will AI x Bio cure diseases? Unequivocally yes, imo… but not by the route that gets the most airtime. Rather than timeline acceleration & compression, I see orchestration & parallelization as today's low-hanging fruit. Most drug hunters see a handful of go/no-go calls in a career, & spend entirely too much of the rest trading emails (my inbox can attest). Give them the bandwidth to run several programs at once, and the time to focus on scientific judgment, and I’m willing to bet they’ll make better calls. And at scale, better calls translate to more drugs, higher hit rates, broader patient impact, & likely, a disease or 10 or 100 cured. I couldn't be more excited to build toward that future @AnthropicAI. Finally, a heartfelt thank you to BioTwitter/X. The end of anonymity is bittersweet, but if anyone’s earned a proper introduction, you all have. It’s difficult to overstate how much this app has done for me: a decade of education, plenty of friends, many job opportunities, several mentors, a few investors, even a co-founder (fmr). It was the generosity of this community that inspired me to start posting 6 years ago, after just as many years lurking. I hope I’ve been able to pay it forward, in my own small way. But, to be clear, Prof Oak isn’t going anywhere. Same beat, same themes, and now, a front-row seat to drug development at the frontier. Stay tuned for dispatches from the trenches; and remember... Gotta Cure 'Em All

10 months later, frontier models are approaching 75% accuracy on SpatialBench. GPT-6.1 Terra + Pi sets a new high at 73.3%. When we released this, it wasn’t obvious that agents would become reliable at measurement-specific bioinformatics, let alone this quickly. Programmers were already adopting these tools, but much of biotech was still unfamiliar with the idea that you could point a coding agent at experimental data and use it to answer scientific questions. The chunked step structure made sense in that context. Errors compounded quickly across end-to-end analysis workflows. Normalization, clustering and cell typing each needed focused treatment, with choices informed by how the assay works. Imaging-based methods (MERFISH, Xenium, CosMx) bring different analysis challenges than sequencing-based methods (Visium, Slide-seq). We wanted to understand where agents could reliably handle these details and where they still failed. For the strongest frontier models, longer research workflows, under-explored modalities and experimentally verified decisions are becoming more interesting targets for evaluation. The increase in benchmark scores is itself exciting, bringing us closer to widespread, automated bioinformatics analysis of spatial data. We believe spatial biology, which is going through its own revolution, has a special, reciprocal relationship with agentic data analysis. Richer measurements give agents more to work with. Better analysis lets scientists ask more questions, understand more biology and run more experiments, consuming more kits and compute along the way. Progress in biology is moving quickly.

2026 will be the year of agents in biology. But we need better benchmarks. We worked with scientists to turn real world analysis into verifiable problems. SpatialBench stratifies frontier models, shows harnesses matter, and reveals distinct failure modes between model families:

Ventured to the Genentech campus to share principles of biology task design and how benchmark shape changes across multi-omics, biosecurity and therapeutics. Wonderful, curious scientists thinking very seriously about these problems. Benchmarks will be a first-class tool in this new ecosystem.

Thank you to @zoe_piran + Arindam Sett for hosting

Gemini 4 Argon quite strong at antibody (4th, 63.0%) and preclinical pharmacology (3rd, 64.7%) evals

Think a lot of routine CRO work was always likely to move offshore. Wouldn't underestimate WuXi either. Likely a durable part of the biotech ecosystem. Optimistic that American scientific and engineering expertise will produce a different kind of CRO in response. Strong robotics/AI talent working alongside scientists, with full control over experimental infrastructure. Lots of room to change how useful experiments are run and data is generated.

On #2: Andy Grove once said “abandoning today's 'commodity' manufacturing can lock you out of tomorrow's emerging industry” and nowhere is this more true than in pharma and life sciences. Over the last 20 years we have shifted much of the drug discovery process to large Chinese contract research organizations (CROs) like Wuxi—a company which was added to the DoW’s list of “Chinese military companies” in June. We’ve seen this multiple times now. When the work moves overseas, the expertise and infrastructure follow. We're at risk of losing the ability to turn our ideas into medicines. That’s an unacceptable dependency for something so central to our health and our future.

Think many underestimate how much progress we can make in wet lab automation with sufficient creativity + focused engineering using existing tech (eg. commodity semiconductor components). Barriers to progress in this category have been fragmentation in both market and lab workflows. The reason for both is similar: lots of tasks in biology are bespoke and this is immediately clear from looking at the methods + supps of papers that on the surface present similar results. If you let loose twenty insane engineers on the problem of moving sequencing prep to a chip, the reward for this effort has been uncertain. But this is changing. One happy accident of improving molecular measurement tools, like spatial and single cell, is the consolidation of previously fragmented prep steps into increasingly standard 'core workflows'. One way to see this is to audit the behavior of scientists at large labs, break down how much true variability exists between tasks and which are actually of the same shape. This standardization in procedural lab work will do much to move the automation needle.

Long on miniaturizing wet lab workflows onto chips. A building’s worth of experimental capacity, programmable in software.

RT @anshulkundaje: This class of models need drastically different evaluation schemes and I'm not seeing serious efforts in this space. Good area for serious eval research cc @LatchBio

We published results for Claude through LSVP on our therapeutics benchmarks. Considerable uplift, especially for Opus 5.5, on tasks drawn from small molecule, antibody and oligonucleotide development programs.

Our current benchmarks focus on domain-specific bioinformatics, therapeutics capabilities or biosecurity task categories (you can find the full herd at While domain-specific benchmarks provide rigorous treatment of complex areas of science and help organize failure modes, we believe that we have now measured enough data types to warrant a meta-benchmark useful as a high-level barometer on model progress. We introduce OmicsBench v1, a collection of 784 tasks from eight benchmarks whose primary concern is the analysis of molecular (“omics”) data: SpatialBench, ScBench, EpiBench, VariantBench, BioSecBench-Surveillance, BioSecBench-Function, SpatialBench-Long, and ScBench-Long. Chris wrote an awesome essay below:

Introducing OmicsBench v1 - our attempt to measure -omics capabilities and analyze existing benchmarks on new models. More to follow feat. @arjunomics, @harm0n, @Harihara_subrah, @evanseeyave, @DianzhuoWang, @lehannahle, @tpr720, @kenbwork, and many more

We’ve released quite a few frontier biology benchmarks this year. While focusing on individual scientific domains provides rigorous treatment of complex task categories, we have now covered enough ground to take a broader view of model progress. Here we introduce OmicsBench v1: 784 tasks from eight prior benchmarks focused on analyzing molecular data, covering spatial omics, single-cell transcriptomics, epigenomics, variant analysis, microbial surveillance and protein function. Domain experts constructed these tasks by analyzing published experimental datasets, reproducing reported results or generating new findings. The collection includes 741 short-horizon tasks covering individual analysis steps and 43 long-horizon tasks covering complete workflows. For this release, we added further human verification, revised prompts and research environments, disabled internet access, and removed additional identifying information from datasets. We tested 31 model-harness combinations. The strongest short-horizon configurations from xAI, OpenAI and Anthropic scored similarly: Grok 4.7 at 49.4%, GPT-6 Astra at 49.3% and Opus 5 at 47.7%, with overlapping confidence intervals. GPT-5.6 Sol with Codex led on long-horizon tasks at 32.8%. Native harnesses gave no large advantage on average. Agents have gotten better at omics analysis. Most of the improvement we measured came between February and July, with smaller gains in subsequent releases. No later model we tested exceeded GPT-5.6 Sol on long-horizon tasks. Agents also continue to make mistakes. They apply conventions from one assay to another, choose inappropriate scales for cellular neighborhoods, and mistake background microbial sequences for signal. Across domains, we see agents picking thresholds without validation, ignoring their own results and inaccurately reimplementing tools. We also see attempts to identify the studies behind sanitized datasets, retrieve benchmark answers and reverse-engineer the outputs a grader expects. Combining these benchmarks has given us a chance to revisit earlier tasks with more capable models and look closely at how their behavior has changed. It has also raised questions we want to explore: 1/ Models may solve problems differently from humans. Is it desirable for models to memorize biological studies and reference information, such as genomic coordinates? 2/ When we cannot see a model’s full thinking trace, how often are we missing attempts to game the task? 3/ How can we design evaluations that require models to provide interpretable scientific traces? 4/ How can we reduce contamination while retaining the real-world relevance of our data? We are excited to try benchmark construction ideas from other efforts, including GeneBench-Pro, Terminal-Bench Science, RareBench and BioMysteryBench. Benchmarking + evaluation is a rapidly evolving and challenging category of engineering. Biology provides a rich landscape of knowledge and its own interesting challenges for this work. With wet lab verification and drug development programs in the pipeline, we’re excited to bring these benchmarks closer to doing science. The future of biology comes from engineers and scientists working closely together. More to come.

- - -