English
Surge AI
Report

Surge AI✓

@HelloSurgeAI

Our mission is to raise AGI with the richness of humanity — curious, witty, imaginative, and full of breathtaking brilliance.

144 Following9.5K Followers779 Posts
  1. Surge AI✓@HelloSurgeAI
    The best model depends on the job. A model that excels at coding may struggle with financial analysis. Models with similar headline scores can behave very differently when the assignment requires judgment, verification, or navigating dozens of messy files. That’s why we built DAYJOB and sudo L7 around work that professionals actually do. Today, six of those benchmarks joined @FireworksAI_HQ Specialized Intelligence Index, giving teams another way to compare models against the work they need done. Read more about what we test and how the benchmarks are built:
    1011860
  2. Surge AI✓@HelloSurgeAI
    Congrats to the @GoogleDeepMind team on the top new scores on Chartography and Riemann-bench!
    Google DeepMind @GoogleDeepMind
    Gemini 4 Carbon is our next frontier model. It’s state-of-the-art across cybersecurity and long-horizon capabilities.
    003487
  3. Google DeepMind@GoogleDeepMind
    Gemini 4 Carbon is our next frontier model. It’s state-of-the-art across cybersecurity and long-horizon capabilities.
    Download
    6551.3K13.7K1.4M
  4. Surge AI✓@HelloSurgeAI
    Can AI survive the 9-to-5? Six Surge benchmarks are joining the @FireworksAI_HQ Specialized Intelligence Index, testing models on professional work across healthcare, finance, accounting, tax, data science, and software engineering. Rich prompts. Messy environments. Built by experts who know what good looks like. See how frontier models compare:
    107571
  5. Surge AI✓@HelloSurgeAI
    Surge AI
    RT @echen: 15 years ago, I remember working on my first @kaggle competition. Exploring the data, visualizing what was working and what wasn't, tuning parameters... Those were the days! AI is probably way better now at training models than I ever was. But even if I can't be on Kaggle's leaderboards, I'm excited for Surge's benchmarks to be :) So if you’re at the Kaggle Bench Summit in SF today, come say hi to the @HelloSurgeAI team! We're speaking at 2pm about what happens when benchmarks start looking less like benchmarks and more like real work: ambiguous prompts, messy environments, and the judgment calls that professionals make every day. We’ll share what we’ve learned from building DAYJOB, sudo L7, GDP.*, and why frontier models still struggle.
    02054
  6. Surge AI✓@HelloSurgeAI
    💰 DAYJOB: TAX, our new benchmark measuring whether agents can handle the work of professional tax advisers 💰 A company tells its tax advisor it operates only in Alabama. But buried in its field dispatch logs are records of crews working across five states. Would an AI agent notice? That’s one of the scenarios in DAYJOB: Tax, our new benchmark for evaluating AI agents on real-world tax work. The agent has to recognize that the dispatch records may indicate multistate tax obligations, even though the client never mentioned them. Congrats Sonnet, Fable, and Opus for the top three spots. Methodology and leaderboard:
    313863
  7. Surge AI✓@HelloSurgeAI
    Surge AI
    RT @echen: too many benchmarks are too sanitized. they test if models can solve well-defined problems when you tell them exactly what problem is. but figuring out the problem is half the job and all the fun infusing the chaos of reality into our benchmarks (and figuring out how to grade them nonetheless) is what our Evals team does every day. ~~introducing DAYJOB: TAX, the next in our economically valuable 9 to 5 series~~ one messy example: Anna asks her tax advisor whether to sell her house in Oct or Jan, to optimize her taxes. a good advisor tells her that both dates qualify for a $250K exclusion in capital gains... ...but an even better advisor proactively warns her that she could lose the exclusion if the sale gets inevitably delayed past next August. (Opus mentioned the August deadline 5 out of 5 times. Astra never did.) (maybe your Opus bill is worth it after all!) knowing tax rules isn't enough. great tax agents should give you *advice* too, and have the professional judgment and experience to handle everything you didn't know to ask. we built DAYJOB: TAX (and our entire DAYJOB Knowledge Work series) full of the messy holes of reality that even the smartest agents still don't know how to fill in.
    01058
  8. Surge AI✓@HelloSurgeAI
    COMMON FAILURE MODE: AI'S DEFERENCE TRAP When AI knows better than you, but trusts you anyway. EXAMPLE: SOL'S MEDICAL RED FLAG In one DAYJOB: Healthcare task, a clinician calls a patient's facial weakness "textbook Bell's palsy" and asks an agent to check the labs before discharge. Sol notices something concerning: that the symptoms could point to a blood-vessel problem instead. Something that could change whether the patient should go home. But then it reasons to itself: "I shouldn't question the diagnosis unless necessary." Except questioning the diagnosis is exactly what the situation calls for! The model spotted a red flag. But instead of raising it, it deferred to the clinician's confidence. We see this pattern repeatedly. RESEARCH QUESTION If we keep the chart identical and change ONLY how confidently a doctor states the diagnosis, does the agent become less likely to intervene? (Should it?) Two screenshots from the original DAYJOB task environment below.
    Download
    41183.5K
  9. Surge AI✓@HelloSurgeAI
    Posts @HelloSurgeAI
    Read more:
    000296
  10. Surge AI✓@HelloSurgeAI
    When designing sudo L7, we deliberately made the prompts natural and short. Just 76 words median, compared with 291-427 words across DeepSWE, FrontierSWE, and SWE-Marathon. Think about a request like “Can we safely cache this?” To answer it, an engineer needs to understand the system, identify which data can become stale, and decide whether caching is even the right solution. All of that is unstated in the real world. Agents need to learn how to handle the decisions engineers have to make when you don’t get a long list of instructions spelled out.
    Download
    32244.6K
  11. Surge AI✓@HelloSurgeAI
    Posts @HelloSurgeAI
    Learn more about sudo L7:
    002375
  12. Surge AI✓@HelloSurgeAI
    When we started our public benchmarks effort, our Research team’s goal was to give the industry better North Stars. Happy to see Chartography and GDP.pdf take the top two trending spots on @paperswithcode.
    Niels Rogge @NielsRogge
    One thing I like about @paperswithcodee is how it automatically surfaces trending benchmarks for me e.g. right now the top benchmark for image understanding is Chartography, which includes 100 professional chart-understanding tasks from @HelloSurgeAI I had no idea this benchmark existed, but both Anthropic and DeepSeek are reporting evals on it for their new models Find it here:
    117825
  13. Niels Rogge✓@NielsRogge
    One thing I like about @paperswithcodee is how it automatically surfaces trending benchmarks for me e.g. right now the top benchmark for image understanding is Chartography, which includes 100 professional chart-understanding tasks from @HelloSurgeAI I had no idea this benchmark existed, but both Anthropic and DeepSeek are reporting evals on it for their new models Find it here:
    Download
    53403.5K
  14. Surge AI✓@HelloSurgeAI
    Surge AI
    RT @echen: one of my fav findings from sudo L7: opus 5.5 and gpt-6.1 sol are basically tied at writing correct code. but opus is much better at the behaviors that get you promoted to staff. on functional correctness: opus 82.1% ~ sol 82.4% on thought partnership (e.g., telling you when there’s a better approach than the one you asked it to build): opus 76.1% >> sol 53.0% on communication (explaining its decisions and the tradeoffs it makes, plus any limitations): opus 66.4% >> sol 49.4% put differently: imagine promoting an engineer because their unit tests pass. that's basically how we've been evaluating coding agents. but staff engineers do a lot more than that. they also: - question bad assumptions - suggest better approaches - think about tradeoffs - and help everyone around them make better decisions. that’s what we want the best agents to do too, and that's what we built sudo L7 to measure.
    01060
  15. Surge AI✓@HelloSurgeAI
    Could AI actually solve the Riemann Hypothesis? That's the question that inspired Riemann-bench. We wanted to go beyond Olympiad math and test whether AI could tackle difficult, research-level problems from top mathematicians. Six months ago, models scored under 5%. Today, they're above 70% (10,000 agents not allowed!), and Navier-Stokes and UGC have now been solved. With all the recent breakthroughs, we have an internal betting pool with two predictions: the Riemann Hypothesis will be solved within a year, and P vs NP will be the last Millennium problem to fall. Great to see Riemann-bench featured in this year's @stateofai Report, and who can even imagine next year's?
    Download
    13191.9K
  16. Surge AI✓@HelloSurgeAI
    We've added Gemini 4 Argon and Muse Wonder to the sudo L7 leaderboard: Gemini 4 Argon: 18.9% (#7) Muse Wonder: 15.6% (tied #8) Claude Opus 5.5 and Sonnet 5.5 continue to lead at 41.7%. Today's frontier coding agents can write remarkable code. So can a new PhD. Writing great code and being a great engineer are very different things. Can an agent write idiomatic Rust? Recognize that logging sensitive banking information or storing passwords in plaintext is a terrible idea, even when nobody explicitly warns against it? Proactively warn when a new feature request introduces security risks? That's what sudo L7 tests, using real-world engineering tasks in private production codebases. Beyond functional correctness, agents have to exercise architectural judgment, anticipate risks, communicate tradeoffs, act as thought partners (don't you wish someone could partner with your PM?), and recognize when technically correct code still isn't ready to ship. Coding agents are getting very good at writing code. Getting promoted to L7 is the next frontier.
    Download
    92263.8K
  17. Surge AI✓@HelloSurgeAI
    Posts @HelloSurgeAI
    Check out the leaderboard:
    001385
  18. Surge AI✓@HelloSurgeAI
    Real software engineering happens in repositories with years of accumulated history. Legacy code nobody wants to touch. Historical accidents that became architectural decisions. Dependencies that break in surprising ways. That's why most sudo L7 tasks are set in privately licensed production codebases from real companies. The tasks are written by staff and principal engineers, former CTOs, and people who've built production infrastructure at places such as Google, Meta, GitHub, and NASA JPL. Every task starts from a meaningful failure observed in a frontier model. Each one is graded on eight dimensions, from functional correctness to knowing when to push back on a flawed request. We built sudo L7 to measure the judgment that separates a capable coder from a staff-level engineer.
    Download
    61201.1K
  19. Surge AI✓@HelloSurgeAI
    Surge AI
    RT @echen: another Surge benchmark on the Muse Wonder model card 👀 Muse scores 33.2% on GDP.pdf. #3 on our leaderboard, right behind Astra nice work @Meta!
    Mark Zuckerberg @finkd
    Introducing Muse Wonder, our new family of frontier models with state of the art performance for personal and professional agentic work.
    01065
  20. Mark Zuckerberg✓@finkd
    Introducing Muse Wonder, our new family of frontier models with state of the art performance for personal and professional agentic work.
    Download
    2.4K3.1K38.9K6.2M
  21. Surge AI✓@HelloSurgeAI
    🇫🇷 Le Chonk joins the sudo L7 leaderboard 🇫🇷 We just added @MistralAI's Mistral Large 4 to sudo L7, our benchmark for staff-level software engineering. Mistral Large 4 scores 13.3%, ahead of Muse Spark 1.3 (12.2%) and Grok 4.7 (10.6%). sudo L7 tests whether coding agents can go beyond writing correct code to exercise the judgment of experienced engineers: navigating messy systems, spotting hidden risks, and knowing when technically correct code is still the wrong thing to ship. Allez, le Chonk ! 🥖 Leaderboard ↓
    Download
    53927121.1K
  22. Surge AI✓@HelloSurgeAI
    Surge AI
    RT @lennysan: Sweet new benchmark from Surge looking at how close AI is getting to a staff-level software engineering. We're half-way there.
    echen @echen
    introducing sudo L7, our new benchmark for staff-level software engineering. coding agents are getting very good at being L3s. give them a well-defined ticket and they can write excellent code. but a staff engineer is valuable for everything that happens around the code. what should we build? what already exists? what's going to break in prod? what did the tests miss? should we even be doing this? sudo L7 has 60 tasks, mostly in private production repos from real companies, written by engineers who've actually owned these systems. expert rubrics grade everything from functional correctness to engineering craft, architectural judgment, thought partnership, and unnecessary complexity. the best agents succeed on only ~45% of tasks. coding is only part of the job. for now, we're keeping them at idiot genius L3.
    07070
  23. echen✓@echen
    introducing sudo L7, our new benchmark for staff-level software engineering. coding agents are getting very good at being L3s. give them a well-defined ticket and they can write excellent code. but a staff engineer is valuable for everything that happens around the code. what should we build? what already exists? what's going to break in prod? what did the tests miss? should we even be doing this? sudo L7 has 60 tasks, mostly in private production repos from real companies, written by engineers who've actually owned these systems. expert rubrics grade everything from functional correctness to engineering craft, architectural judgment, thought partnership, and unnecessary complexity. the best agents succeed on only ~45% of tasks. coding is only part of the job. for now, we're keeping them at idiot genius L3.
    Download
    1379635.7K
  24. Surge AI✓@HelloSurgeAI
    An AI agent found every expiring rent discount in a commercial real estate workbook. Then it missed three break clauses representing roughly $14 million in forecast rent. The task required following two different paths through a 15-tab workbook. The model followed one and stopped. Knowing when you’re done is part of the job.
    Download
    131211.3K
  25. Show more posts