AI模型在不同模态下是否能进行类人抽象推理？
Do AI Models Perform Human-like Abstract Reasoning Across Modalities?
AI总结：
本文研究了AI模型在不同模态下进行抽象推理的能力，发现其在文本和视觉模态中的表现存在显著差异，表明仅依赖准确率评估抽象推理能力存在偏差。
摘要：
OpenAI的o3-preview推理模型在ARC-AGI-1基准测试中超过了人类的准确率，但这是否意味着最先进的模型能够识别并运用基准测试所设计的抽象能力？在这里，我们通过密切相关但更简单的ConceptARC基准测试来调查AI模型的抽象能力。我们的评估变化了输入模态（文本与视觉），外部Python工具的使用以及推理努力。除了输出准确率之外，我们还评估了模型生成的自然语言规则，以确定模型是否识别出ConceptARC所设计的抽象能力。我们发现，最佳模型的规则往往基于表面的“捷径”，捕捉预期的抽象能力远不如人类。在视觉模态中，AI模型的输出准确率急剧下降；然而，我们的规则层面分析显示，其大部分规则捕捉了预期的抽象能力，尽管模型在应用这些概念生成正确解决方案时存在困难。简而言之，我们表明，仅使用准确率来评估抽象推理可能会在文本模态中高估AI的能力，在视觉模态中低估其能力。我们的结果提供了对AI模型抽象推理能力更真实的图片，并为朝着类人、以抽象为中心的智能迈进提供了更系统的方法。
英文摘要：
OpenAI's o3-preview reasoning model exceeded human accuracy on the ARC-AGI-1 benchmark, but does that mean state-of-the-art models recognize and reason with the abstractions the benchmark was designed to test? Here we investigate abstraction abilities of AI models using the closely related but simpler ConceptARC benchmark. Our evaluations vary input modality (textual vs. visual), use of external Python tools, and reasoning effort. Beyond output accuracy, we evaluate the natural-language rules that models generate to explain their solutions, enabling us to assess whether models recognize the abstractions that ConceptARC was designed to elicit. We show that the best models' rules are frequently based on surface-level ``shortcuts,'' capturing intended abstractions considerably less often than humans. In the visual modality, AI models' output accuracy drops sharply; however, our rule-level analysis reveals that a substantial share of their rules capture the intended abstractions, even as the models struggle to apply these concepts to generate correct solutions. In short, we show that using accuracy alone to evaluate abstract reasoning can substantially overestimate AI capabilities in textual modalities and underestimate it in visual modalities. Our results offer a more faithful picture of AI models' abstract reasoning abilities and a more principled way to track progress toward human-like, abstraction-centered intelligence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
