François Chollet, an A.I. researcher, created a puzzle game called ARC in 2019 that was easy for humans but challenging for machines. This game was designed to track A.I. progress and push back against claims that machines would outsmart humanity. The puzzles consist of identifying visual patterns based on a few examples, which A.I. systems struggled to do for years.
In December, OpenAI announced that its A.I. system, o3, had surpassed human performance on Chollet’s test. While this was seen as a step towards artificial general intelligence (A.G.I.), critics pointed out the limitations of benchmark tests like ARC. Arvind Narayanan, a computer science professor, questioned whether the test truly measured progress towards A.G.I.
Chollet and Mike Knoop launched the ARC Prize, offering $1 million to anyone who could beat human performance on the benchmark. Despite over 1,400 submissions, no winner emerged, with OpenAI’s o3 scoring the highest but not eligible due to cost and lack of technology sharing.
The ARC Prize recently introduced a more challenging benchmark, ARC-AGI-2, offering puzzles that even OpenAI’s system struggled with. Chollet emphasized the importance of machines efficiently acquiring new skills, a gap that still exists in A.I. technology. The ARC Prize Foundation aims to continue developing benchmarks to measure intelligence and may debut ARC-AGI-3 in 2026, focusing on dynamic, real-world interactions.
As A.I. technology advances, the goal posts for measuring intelligence will shift, potentially leading to the achievement of artificial general intelligence. Chollet believes that true A.G.I. will be reached when machines can master tasks that are easy for humans but impossible for A.I. systems.
Source
Note: The image is for illustrative purposes only and is not the original image of the presented article.
