🚀Excited to release our study on visual comprehension and abstract reasoning skills via
#PolyMATH. We provide both quantitative and qualitative evaluations of
#GPT4o,
#Claude 3.5 Sonnet,
#Gemini 1.5 pro,
#OpenAI o1 models & 13 other models.
📄✨Full paper:
🤗
@huggingface Dataset:
🔍 Key Insights:
1️⃣ A dataset of 5000 samples to test cognitive reasoning capabilities of MLLMs.
2️⃣ The best scores achieved on POLYMATH are ∼ 41%, ∼ 36%, and ∼ 27%, obtained by Claude-3.5 Sonnet, GPT-4o and Gemini-1.5 Pro respectively while human baseline was at ~66%.
3️⃣ An improvement of 4% is observed when image descriptions are passed instead of actual images, indicating reliance on text over image even in multimodal reasoning.
4️⃣ Open AI o1 models get competitive scores with human baseline on text only samples, highlighting room for improvement!
A massive shoutout to our outstanding team
@s_verma3011,
@ujjwala_ananth,
@thegraydleguy and
@Mihir3009 and guidance of
@Swarooprm7 and
@cbaral