LatchBio
RT @chriswzou: Our benchmarks represent the collective imagination of every scientist at Latch for what LLMs could do in the future. Awesome to see this out and everyone's work recognized
03020
arjun @arjunomics
Kenny Workman @kenbworkVentured to the Genentech campus to share principles of biology task design and how benchmark shape changes across multi-omics, biosecurity and therapeutics. Wonderful, curious scientists thinking very seriously about these problems. Benchmarks will be a first-class tool in this new ecosystem.
lada @ladanuzhnaCo-hosting a dinner with @kenbwork next Tue (10/27) for folks doing research at the intersection of AI × Bio! Wish list for topic distribution: gLMs & seq2func, regulatory variant-to-gene mapping, de novo protein/RNA design, AI-guided experimental design & benchmarking models on practical scientific work Would love to meet more people actively working on these! Reg:
Christopher Zou @chriswzouIntroducing OmicsBench v1 - our attempt to measure -omics capabilities and analyze existing benchmarks on new models. More to follow feat. @arjunomics, @harm0n, @Harihara_subrah, @evanseeyave, @DianzhuoWang, @lehannahle, @tpr720, @kenbwork, and many more
lada @ladanuzhnaCo-hosting a dinner with @kenbwork next Tue (10/27) for folks doing research at the intersection of AI × Bio! Wish list for topic distribution: gLMs & seq2func, regulatory variant-to-gene mapping, de novo protein/RNA design, AI-guided experimental design & benchmarking models on practical scientific work Would love to meet more people actively working on these! Reg:
Ramesh K Ramasamy @ramesh8vOur multi-omics exercise paper is out in Cell! We all know exercise is good for us. But what actually happens at the molecular level when we exercise regularly? In this study, we set out to understand this by analyzing bulk RNA-seq data from 22 mouse tissues after 8 weeks of voluntary exercise. We also generated single-nucleus RNA-seq, single-nucleus genome-wide DNA methylation, proteomics, and metabolomics data to investigate tissue-, sex-, and time-of-day-dependent responses. Some interesting findings: 1. We observed widespread molecular changes across multiple organs, with responses varying by tissue, sex, and time of day. 2. By profiling ~36,000 single-nucleus methylomes alongside ~268,000 single-nucleus transcriptomes, we identified cell-type-specific DNA methylation changes associated with muscle adaptation that bulk measurements cannot resolve. 3. We found that males and females respond differently to exercise, with distinct transcriptional and DNA methylation signatures. 4. We identified a 51-gene signature that distinguishes exercise-trained from sedentary states across independent datasets, including human and rodent studies. 5. We identified 180 differentially expressed transcription factors and 4,633 predicted target genes, revealing cell-type-specific regulatory networks associated with muscle adaptation to exercise. Beyond these findings, we've made the datasets available as a resource for the research community to explore the molecular mechanisms of exercise. It was a fantastic collaborative effort across multiple labs and disciplines. Thanks to everyone involved in making this work possible! Paper: #MultiOmics #Exercise #Epigenetics #SingleCell
Kenny Workman @kenbworkWe introduce OncologyBench, a verifiable benchmark of 25 multistep oncology evaluations that tests the ability of multi-agent systems to produce clinically relevant decisions in cancer. 25 evaluations, built on 92 component tasks, span six stages of an oncology journey: (1) Translational and basic research (2) Risk and detection (3) Diagnosis and tumor characterization (4) Treatment (5) Response and resistance (6) Outcome Evaluations draw on published datasets, case studies, public databases, and patient data graciously donated to Latch for the express purpose of benchmarking. Clinical and translational decisions in oncology are rarely made by one person working from one dataset. They are built on several analyses, each carried out by a specialist on a given data type, whose reports are then weighed by someone who does not typically repeat the underlying work. OncologyBench is built to benchmark agents working in that setting. Agents received experimental data and worked without internet access. The strongest of the 31 model-harness combinations we tested were Claude Opus 5, Opus 5.5 and Sonnet 5.5, passing 29 to 40% of attempts, followed by Kimi K3 at 24%. We found that the models with the strong interpretation and analysis skills still showed weakness in reasoning. Two-thirds of the attempts that got every analysis right still failed the final decision. In a separate experiment we found decision and answer quality varied depending on the persona of the question asker. An example evaluation: A 76-year-old woman has undergone resection of stage II mucinous cecal cancer. The evaluation asks whether she should receive adjuvant chemotherapy. Five specialist agents analyze her tumor mutations, pharmacogenomics, gene expression, T-cell infiltration and clinical record. A decision agent receives their reports without access to the underlying data. The tumor is MSI-high, which predicts no benefit from 5-FU monotherapy. CDX2 loss and low T-cell infiltration do not establish that chemotherapy would improve her outcome. DPYD*2A is detected in 6 of 58 tumor-sequencing reads, but these data neither confirm nor exclude an inherited variant associated with severe 5-FU toxicity. BRAF V600E and MLH1 silencing support a sporadic origin. The expected recommendation is observation, with missing pathology requested and germline DPYD testing before any future 5-FU. --- Incredible work from researchers + former biotech founders at Latch. Gratitude to the patients that donated their data. More to come.
Kenny Workman @kenbworkHiring for Members of Technical Staff to work on rigorous evaluation of AI in biology. Our mission is to advance humanity's ability to engineer biology. We approach this by understanding and improving AI’s application to biology across basic research, drug discovery, and biosecurity. We are a team of engineers, scientists, and operations specialists who are driven by improving capabilities and interpretability. Some of our work is showcased on About this role You’ll turn your understanding of a biological domain into a roadmap of benchmarks and design them with engineers. Then build and manage your own scientific team using your network and internal talent infrastructure. You have (1) deep scientific expertise and experience using AI tools (2) or strong coursework in math/CS and experience working with scientists This is a high-intensity, founder-type role. Our Members of Technical Staff have built companies, run drug programs and scaled complex technical projects. Projects you might work on - Design benchmarks with engineers and scientists, from selecting scientific questions and datasets to developing reliable graders. - Work with engineers on large-scale computing and AI inference infrastructure to evaluate models and investigate their failures. - Research new methods for evaluating scientific capabilities and measuring improvements from training. - Work with our internal data generation and drug development teams to build tasks around real experiments and research decisions. How we work Our MTS team works in person in Mission Bay, San Francisco. We keep meetings to a minimum and are flexible about when you arrive or leave. Tight deadlines and customer commitments sometimes require late hours. Our work in use Frontier AI labs use our benchmarks to assess biological capabilities and safeguards. Our work appears in research announcements and model cards, including: - - - - - Hiring Process Our process is typically completed within two weeks. - Round 1: Introduction - Round 2: Technical project matching - Round 3: Take-home evaluation project (~2 hours) - Round 4: Technical interview / live engineering session (1 hour) - Round 5: Paid onsite project and presentation (2–3 days) - Offer Compensation & Benefits - 350K-500K+ OTE - Equity - Unlimited PTO - Health, dental and vision insurance fully covered - Visa sponsorship and relocation assistance - 401(k) - Office in Mission Bay, San Francisco - Lunch and dinner provided Apply at Or send an email to kenny[at]latch[dot]bio
Kenny Workman @kenbworkWe introduce OncologyBench, a verifiable benchmark of 25 multistep oncology evaluations that tests the ability of multi-agent systems to produce clinically relevant decisions in cancer. 25 evaluations, built on 92 component tasks, span six stages of an oncology journey: (1) Translational and basic research (2) Risk and detection (3) Diagnosis and tumor characterization (4) Treatment (5) Response and resistance (6) Outcome Evaluations draw on published datasets, case studies, public databases, and patient data graciously donated to Latch for the express purpose of benchmarking. Clinical and translational decisions in oncology are rarely made by one person working from one dataset. They are built on several analyses, each carried out by a specialist on a given data type, whose reports are then weighed by someone who does not typically repeat the underlying work. OncologyBench is built to benchmark agents working in that setting. Agents received experimental data and worked without internet access. The strongest of the 31 model-harness combinations we tested were Claude Opus 5, Opus 5.5 and Sonnet 5.5, passing 29 to 40% of attempts, followed by Kimi K3 at 24%. We found that the models with the strong interpretation and analysis skills still showed weakness in reasoning. Two-thirds of the attempts that got every analysis right still failed the final decision. In a separate experiment we found decision and answer quality varied depending on the persona of the question asker. An example evaluation: A 76-year-old woman has undergone resection of stage II mucinous cecal cancer. The evaluation asks whether she should receive adjuvant chemotherapy. Five specialist agents analyze her tumor mutations, pharmacogenomics, gene expression, T-cell infiltration and clinical record. A decision agent receives their reports without access to the underlying data. The tumor is MSI-high, which predicts no benefit from 5-FU monotherapy. CDX2 loss and low T-cell infiltration do not establish that chemotherapy would improve her outcome. DPYD*2A is detected in 6 of 58 tumor-sequencing reads, but these data neither confirm nor exclude an inherited variant associated with severe 5-FU toxicity. BRAF V600E and MLH1 silencing support a sporadic origin. The expected recommendation is observation, with missing pathology requested and germline DPYD testing before any future 5-FU. --- Incredible work from researchers + former biotech founders at Latch. Gratitude to the patients that donated their data. More to come.
Kenny Workman @kenbworkWe introduce OncologyBench, a verifiable benchmark of 25 multistep oncology evaluations that tests the ability of multi-agent systems to produce clinically relevant decisions in cancer. 25 evaluations, built on 92 component tasks, span six stages of an oncology journey: (1) Translational and basic research (2) Risk and detection (3) Diagnosis and tumor characterization (4) Treatment (5) Response and resistance (6) Outcome Evaluations draw on published datasets, case studies, public databases, and patient data graciously donated to Latch for the express purpose of benchmarking. Clinical and translational decisions in oncology are rarely made by one person working from one dataset. They are built on several analyses, each carried out by a specialist on a given data type, whose reports are then weighed by someone who does not typically repeat the underlying work. OncologyBench is built to benchmark agents working in that setting. Agents received experimental data and worked without internet access. The strongest of the 31 model-harness combinations we tested were Claude Opus 5, Opus 5.5 and Sonnet 5.5, passing 29 to 40% of attempts, followed by Kimi K3 at 24%. We found that the models with the strong interpretation and analysis skills still showed weakness in reasoning. Two-thirds of the attempts that got every analysis right still failed the final decision. In a separate experiment we found decision and answer quality varied depending on the persona of the question asker. An example evaluation: A 76-year-old woman has undergone resection of stage II mucinous cecal cancer. The evaluation asks whether she should receive adjuvant chemotherapy. Five specialist agents analyze her tumor mutations, pharmacogenomics, gene expression, T-cell infiltration and clinical record. A decision agent receives their reports without access to the underlying data. The tumor is MSI-high, which predicts no benefit from 5-FU monotherapy. CDX2 loss and low T-cell infiltration do not establish that chemotherapy would improve her outcome. DPYD*2A is detected in 6 of 58 tumor-sequencing reads, but these data neither confirm nor exclude an inherited variant associated with severe 5-FU toxicity. BRAF V600E and MLH1 silencing support a sporadic origin. The expected recommendation is observation, with missing pathology requested and germline DPYD testing before any future 5-FU. --- Incredible work from researchers + former biotech founders at Latch. Gratitude to the patients that donated their data. More to come.