The Abstraction and Reasoning Corpus (ARC) is a visual program synthesis benchmark designed to test challenging out-of-distribution generalization in humans and machines. Since 2019, limited progress has been observed on the challenge using existing artificial intelligence methods. Comparing human and machine performance is important for the validity of the benchmark. While previous work explored how well humans can solve tasks from the ARC benchmark, they either did so using only a subset of tasks from the original dataset, or from variants of ARC, and therefore only provided a tentative estimate of human performance. In this work, we obtain a more robust estimate of human performance by evaluating 1729 humans on the full set of 400 training and 400 evaluation tasks from the original ARC problem set. We estimate that average human performance lies between 73.3% and 77.2% correct with a reported empirical average of 76.2% on the training set, and between 55.9% and 68.9% correct with a reported empirical average of 64.2% on the public evaluation set. However, we also find that 790 out of the 800 tasks were solvable by at least one person in three attempts, suggesting that the vast majority of the publicly available ARC tasks are in principle solvable by typical crowd-workers recruited over the internet. Notably, while these numbers are slightly lower than earlier estimates, human performance still greatly exceeds current state-of-the-art approaches for solving ARC. To facilitate research on ARC, we publicly release our dataset, called H-ARC (human-ARC), which includes all of the submissions and action traces from human participants.
Vector Symbolic Algebras for the Abstraction and Reasoning Corpus
Centre for Theoretical Neuroscience(理论神经科学中心);David R. Cheriton School of Computer Science(大卫·R·切里顿计算机科学学院);University of Waterloo(滑铁库大学);Department of Philosophy(哲学系);Department of Systems Design Engineering(系统设计工程系)
The ARC of Progress towards AGI: A Living Survey of Abstraction and Reasoning
TIB - Leibniz Information Centre for Science and Technology(TIB - 科学技术信息研究中心);L3S Research Center, Leibniz University of Hannover(L3S研究所以汉诺威莱布尼茨大学);Dresden University of Technology, Amazon (work done outside of Amazon)(德累斯顿技术大学,亚马逊(非亚马逊工作))
Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction
Texas A&M University(德克萨斯A&M大学);University of Waterloo(滑铁卢大学);UC San Diego(圣地亚哥大学);Stanford University(斯坦福大学);University of Washington(华盛顿大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Verdent AI;Lambda
Tycho: Active Abstraction with Programmatic World Models for ARC-AGI-3
Dresden University of Technology(德累斯顿工业大学);Amazon(亚马逊);TIB – Leibniz Information Centre for Science and Technology(莱布尼茨科学与技术信息中心);Leibniz University of Hannover(汉诺威莱布尼茨大学)