I’m usually skeptical of sensational headlines, and while that’s true here, there are a few findings I found genuinely interesting, matching what we’ve observed working with many of the largest firms.
The human study is 12 accountants across four month-end close tasks. More broadly, the benchmark heavily favors what frontier models are already very good at: processing huge amounts of data, searching across files, and meticulously following instructions.
And whenever models are at or near 100% on a benchmark, I start wondering whether we’re measuring general capability or a task distribution models have become particularly optimized for.
But there were two things I found much more interesting:
First, the authors themselves point out how different the benchmark is from real accounting work. In the real world, accountants have months or years of accumulated client context, can ask questions and clarify ambiguities, and generally work on problems aligned with their expertise.
This gets at something we think about a lot at Accrual: accounting is deeply contextual. The work depends on client history, prior work, firm knowledge, conversations, and data spread across many different systems.
Second, giving the accountants Claude didn’t really help. Accountants using Claude were ~15x slower than Claude alone and slightly less accurate. In several cases, Claude had the right answer and the human + AI workflow still ended up submitting the wrong one.
My takeaway:
There’s a big difference between a model being capable of doing the work and making that capability useful to an accountant. Simply giving every accountant Copilot doesn’t bridge that gap.
If the accountant still has to assemble all the context, explain the task, prompt the model, interpret the answer, and then go do the work, you haven’t actually automated much.
That’s a big part of why we built Arc the way we did. Arc starts with the context of the firm and its clients, works across the systems accountants already use, and does the work itself. The accountant reviews the finished work rather than driving the model step by step.
The models are clearly getting good enough. The interesting problem now is building the context and workflows around them.
That’s what we’re building with Arc:
aden barton @aden_barton
New research: we hired 12 CPAs to complete four realistic accounting tasks, then had AI models from the past few years complete the same work.
As of May 2025, accountants outperformed AI. Now they lose out to today's frontier models, which are near perfect on these tasks. 🧵
