Interactive Reasoning Benchmarks [+we're hiring]
|
// A North Star To AGI A sneak peek at Interactive Reasoning Benchmarks. AGI will be measured through interactive environments. Traditional benchmarks focus on single-turn tasks: answering a question, labeling an image, or filling in the blank. But if we are going to measure human-like general intelligence, we need to go further. Real-world intelligence unfolds over time, with context, memory, and continuous learning. Humans thrive in these messy, multi-step environments. To actually evaluate AI’s general intelligence, we need interactive, multi-turn benchmarks that better capture how intelligence works. Games are a perfect testbed: they have clear rules, clear objectives, and provide feedback after every move, but still require long-horizon planning and adaptation. Sneak Peek: ARC-AGI-3 In the same presentation, we shared a sneak peek of ARC-AGI-3. ARC Prize is building 100 novel 2D game environments to test abstract reasoning in AI. These will be split between a public train set and a private test set. The games will be easy for humans, but still challenging for current AI.
Want to us help build ARC-AGI-3 to evaluate AI in multi-turn environments? We're hiring Python game developers for full-time contract positions. Email your previous experience and game portfolio to team@arcprize.org - see more details.
|