arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.01374cs.AI
收藏

H-ARC:对人类在抽象与推理语料库基准上表现的稳健估计

H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark

Department of Psychology(心理学系)

Solim LeGris, Wai Keen Vong, Brenden M. Lake, Todd M. Gureckis

AI总结:

本文通过1729名人类在完整ARC基准上的评估,得出更稳健的人类表现估计,并发布H-ARC数据集以促进ARC研究。

摘要:

抽象与推理语料库(ARC)是一个视觉程序合成基准,旨在测试人类和机器在具有挑战性的分布外泛化能力。自2019年以来,使用现有人工智能方法在该挑战上取得的进展有限。比较人类与机器的表现对于该基准的有效性十分重要。虽然已有工作探索了人类能在多大程度上解决ARC基准中的任务,但这些工作要么只使用了原始数据集中的部分任务,要么使用了ARC的变体,因此只能提供对人类表现的初步估计。在本工作中,我们通过让1729名人类参与者完成原始ARC问题集中全部400个训练任务和400个评估任务,获得了对人类表现更稳健的估计。我们估计,人类在训练集上的平均表现介于73.3%至77.2%正确率之间,报告的经验平均值为76.2%;在公开评估集上的平均表现介于55.9%至68.9%正确率之间,报告的经验平均值为64.2%。然而,我们还发现,800个任务中有790个至少能被一个人在三次尝试内解决,这表明绝大多数公开可用的ARC任务原则上可以被通过互联网招募的典型众包工作者解决。值得注意的是,尽管这些数字略低于早期估计,但人类表现仍然远超当前解决ARC的最先进方法。为促进ARC相关研究,我们公开发布了名为H-ARC(human-ARC)的数据集,其中包含人类参与者的所有提交记录和动作轨迹。

英文摘要:

The Abstraction and Reasoning Corpus (ARC) is a visual program synthesis benchmark designed to test challenging out-of-distribution generalization in humans and machines. Since 2019, limited progress has been observed on the challenge using existing artificial intelligence methods. Comparing human and machine performance is important for the validity of the benchmark. While previous work explored how well humans can solve tasks from the ARC benchmark, they either did so using only a subset of tasks from the original dataset, or from variants of ARC, and therefore only provided a tentative estimate of human performance. In this work, we obtain a more robust estimate of human performance by evaluating 1729 humans on the full set of 400 training and 400 evaluation tasks from the original ARC problem set. We estimate that average human performance lies between 73.3% and 77.2% correct with a reported empirical average of 76.2% on the training set, and between 55.9% and 68.9% correct with a reported empirical average of 64.2% on the public evaluation set. However, we also find that 790 out of the 800 tasks were solvable by at least one person in three attempts, suggesting that the vast majority of the publicly available ARC tasks are in principle solvable by typical crowd-workers recruited over the internet. Notably, while these numbers are slightly lower than earlier estimates, human performance still greatly exceeds current state-of-the-art approaches for solving ARC. To facilitate research on ARC, we publicly release our dataset, called H-ARC (human-ARC), which includes all of the submissions and action traces from human participants.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑