Analyzing GPT-5.5 & Opus 4.7 with ARC-AGI-3

// Analyzing GPT-5.5 & Opus 4.7 with ARC-AGI-3

This week we went through 160 replays and reasoning traces from OpenAI’s GPT-5.5 and Anthropic’s Opus 4.7 attempting novel, long-horizon environments.

The scores were just one data point, but the interesting story is how they achieved their score.

Today we’re open-sourcing our analysis package:

• GPT-5.5 - 0.43% (Public Replays)​
• Opus 4.7 - 0.18% (Public Replays)​

As we compare the runs of GPT-5.5 and Opus 4.7 we’re able to see they failed in different ways. This is important because aggregate scores alone would hide this distinction. We found 3 common failure modes:

• True Local Effect, False World Model - The models understand which action produced a change, but they fail to translate the effect into a global rule
• Wrong Level of Abstraction From Training Data - The models mistake an ARC-AGI-3 environment for another game
• Solved The Level, Didn’t Learn The Game - Even if a model beat a level, it’s unable to use that reward signal to enforce the correct actions

// ARC Prize Foundation is hiring

Momentum at ARC Prize is building. New benchmarks, partnerships, and now we're growing the team.

• Game Platform Engineering Lead - A senior engineer to own and evolve the game engine and real-time play infrastructure behind the ARC-AGI series.
• Model Testing and Analysis Lead - A technical researcher to own how we evaluate frontier models on the ARC-AGI benchmarks.

$7.5K referral bonus for any hire you recommend.