The second round of #FirstProof is announced:
Glad to see the organizers planning a rigorous process. Most AI benchmarks today still rely on the "honor system". That’s billions of valuation dollars hinging on integrity, even while there are many performance-boosting tricks that are undetectable in results (adding hints to prompts, human intervention, best-of-N, etc). For this reason, one should interpret one-off performances by proprietary models (e.g., on math contests) with a grain of salt–especially when they are carried out after solutions are already posted online.
FirstProof will be the only evaluation based on the natural distribution of research mathematics that guarantees autonomy and transparency of results. It should be considered the gold standard benchmark in the AI4Math space.
3148710.1K
