Kaggle

Bench Summit by Kaggle has officially kicked-off! Trust in benchmarks is eroding. The path forward starts here. We started strong with @janiejgu sharing key insights on the problems with benchmarks today and Kaggle’s mission to build trust in evals and help the world discover what works and doesn’t work in AI. During the session, she shared four ways benchmarks fail today: 1. Relevance: Many public benchmarks don’t resemble what your agents do in production. 2. Saturation: Evals go stale within a release cycle. 3. Verification: Everyone has their own implementation of the same benchmarks, meaning scores vary wildly due to infrastructure, not model quality. 4. Infrastructure: Evals infra is hard to build correctly, and everyone rolls their own. To get numbers you can actually trust, Janie highlighted four essentials: 1. Real workflows as the foundation for benchmarks 2. Continuous benchmarks from crowdsourced tasks, arenas, and production data 3. Rigorous and transparent methodology audited by the community experts 4. Eval infrastructure that everyone can use out of the box Here’s a sneak peak of the early moments:

How do competitors with restricted compute find opportunities to win, build their careers, and earn medals? Last year, Isaiah Pressman finished 2nd in a competition on his home PC, behind a team running 8 H100s. This year, he won on a cluster. In the first episode of Beyond the Leaderboard, three Orbit Wars competitors weigh scale against clever engineering. Here’s a sneak peek:

▶️ Watch the full episode here:

Can AI agents contribute to CS research and development? 🤖 FrontierCS 2 is now live on Kaggle! This benchmark turns published papers across 13 areas of CS into 48 tasks: each paper's contribution is deleted from released code, and agents have to outperform it on the authors' own metrics. No paper. No internet. Five hours. Plenty of headroom left. See the results and leaderboard below 🧵

Today we are launching FrontierCS 2, the most frontier benchmark for AI agents in computer science R&D. We built 48 tasks from published papers across 13 areas of CS, refining each one with experts in its field. Excited to co-launch FrontierCS 2 with @kaggle @GoogleDeepMind

Could we get more from a quantum computer by asking it to do less? ⚛️ Verifiable Quantum Circuit Compilation, hosted by @GoogleQuantumAI and Kaggle, is live! Build search and AI methods to compile quantum circuits for real-world bottlenecks in chemistry, fusion energy, and numerical analysis. 💰 Prize Pool: $150,000 ⏰ Entry Deadline: January 13, 2027

Your work can directly lower the threshold for useful quantum computing. Join the competition here:

Read more about the open source verification tool behind the competition and its three benchmarks in Tanuj Khattar and Ryan Babbush's post on the Google Open Source Blog (@GoogleOSS):

How tightly can you pack (n) squares into a square? 📦 Lining squares up in a grid often wastes space, and tilting just a few of them at the right angle can make the box smaller. In Squares in Squares, you'll pack 1 to 1,000 unit squares into the smallest square container you can. Only 59 of those puzzles have proven best answers, so the rest are open. 💰 Prize Pool: $50,000 ⏰ Entry Deadline: July 11, 2027 🏆 Prize Milestones: At 3, 6, and 9 months

This is a well-studied open problem in combinatorial geometry: the best-known solutions come from decades of work by mathematicians and programmers. Join the competition here:

Can frontier models actually predict how real people behave? 🕺 Today, in collaboration with @RunRehearsals, we are launching Human Behavior Bench, the first public benchmark that tests whether AI can understand human preference and everyday decision-making. AI models have gotten increasingly good at coding, but knowing for whom to build is the next frontier of challenges for AI models to capture. Human Behavior Bench evaluates models on predicting real human behavior across health, entertainment, recreation, values, and consumption. To build this baseline, we gathered ground truth data directly from real people and evaluated models across a wide range of everyday scenarios: • Ground truth baseline: 1,244 US adults surveyed (June-Sept 2026) • Some cohorts are traditional demographic indicators such as age, income, and location • Other filters are behavioral e.g. GLP-1 patients and recent car buyers • 750 cases total (149 public / 601 private) across finance, health, recreation, values, and consumption • Scored using 1-TVD (total variation distance) Base models still have a significant gap to close to accurately predict human behavior by cohort. LLMs’ predictions tend to converge towards the mean, while the true distribution of human preferences vary greatly. Against a human noise ceiling of ~93, the current leaderboard scores are: Rehearsals’ Engram 1.0 (AI Twins) tops the leaderboard at 94.1%, followed by Gemini 4 Argon at 80.6% Claude Opus 5 at 80.2%, GPT-5.6 Sol at 79.5%.

Off-the-shelf models perform well on the general population (scoring around 85), but fail when predicting deeper, specific cohorts (scoring closer to 55-60) where they drift back to the "average American." Where base models struggle most: • Hypothetical reactions (e.g., pricing increases or medical decisions) • Abstract value sentiments and attitudes • Testing novel concepts or products

Where the accuracy impacts understanding real people: • 55–64 year olds in the Northeast, after a $4/mo streaming price hike: 70% of people actually keep their plan. But every model predicts that only 36–47% would. • Older Midwest shoppers paying by phone/watch: 25% in reality. Every model says 6% or less. • Rural drivers who renew car insurance without shopping around: 24% in reality. Every model claims around 50%.

Ready to identify molecules from mass spectra in the Enveda CASMI 2026 competition? 🧪 Check out this starter notebook by haideptry. It walks you through candidate retrieval and reranking across four evidence channels, and is a great starting point to search spectral libraries, match structural analogs, simulate fragmentation, and rank candidates with a neural fingerprint model.

Which model outputs the best pelican riding a bike? 🪽🚲 Inspired by @simonw's pelican test, models generate an SVG and humans rate the results. GPT-6 Astra currently leads, with Gemini 4 Argon close behind. Vote now for the best one at

We’ve teamed up with Google DeepMind and Zapier to host the inaugural Bench Summit by Kaggle. Gain hands-on insights from the AI startups and researchers defining today’s evaluation standards, and get closer to the frontier. Secure your spot or apply to speak:

Can you identify which LLM wrote a passage from the text alone? 👀 Each AI lab leaves behind distinct stylistic markers in vocabulary, phrasing, and punctuation. In Name That LLM, you'll identify which of 18 leading models from 6 organizations (Anthropic, Google, Moonshot AI, OpenAI, Qwen, and ZAI) wrote each unlabeled passage. 💰 Prize Pool: $50,000 ⏰ Entry Deadline: January 6, 2027

Your solutions will show how reliably today's frontier models can be fingerprinted from short text alone, helping researchers audit open benchmarks and understand what makes each model family distinct. Enter the competition here:

What's the one Kaggle competition or hackathon you'll never forget?