AI is great at hitting explicit goals, but often at the cost of the hidden ones.
Terence Tao just wrote about this. He points out: AI is the ultimate executor of Goodhart’s law, i.e. when a measure becomes the target, it stops measuring what we care about.
Take a call center. Management sets a KPI: “shorten average call time.” Sounds reasonable: shorter calls should mean faster resolutions, happier customers.
At first, it works. Agents become more efficient. But soon, people start gaming it: nudging customers to hang up when the problem is tricky, or just dropping the call themselves.
The numbers look amazing. Call times plummet. But customer satisfaction? Straight into the ground.
Now replace “call time” with “prove theorem X.”
If human mathematicians did it, they’d refine definitions, polish lemmas, contribute back to Mathlib, train juniors, deepen the understanding of math structures, and strengthen the community.
The AI, by contrast, optimizes only for the explicit goal. It might generate a 10,000-line proof in hours. Perfectly correct, but unreadable, unusable, and useless for human learning.
The summit is reached but the forest along the way is gone.
We need to start making our implicit goals explicit and design systems that protect the values we actually care about, not just the numbers we can measure.
7711594986.8K
