A2RBench: 一个用于形式可验证抽象推理基准生成的自动范式
A2RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
AI总结：
本文提出A2RBench自动范式，通过生成、扩展、评估和分析流程提升抽象推理基准生成效率，发现当前LLM在抽象推理能力上存在根本缺陷，且高信息复杂度输入可简化推理过程。
摘要：
抽象推理能力反映了LLM提取和应用抽象规则的智能和泛化能力。然而，准确测量这一能力仍然具有挑战性：现有基准要么依赖昂贵的手动标注，限制了其规模，要么有风险测量记忆而非真正的推理。为此，我们引入了一个名为A2RBench的自动化流程，包括生成、扩展、评估和分析。具体而言，在生成阶段，LLM创建多样化的任务，要求真正的推理；在扩展阶段，LLM重用已验证的规则并扩展新的输入空间以生成任务变体，实现扩展。然而，这一过程可能导致幻觉。为消除它，我们进一步建立了理论框架并证明，程序验证——测试逆操作是否完美地逆转正向操作（循环一致性）——保证了唯一解。通过在主流LLM上的广泛评估，我们发现：（1）当前LLM在抽象推理上存在根本缺陷，顶级模型在代表性子集上显著低于人类（39.8% vs. 68.5%）。（2）当前LLM在生成3D任务的复杂度上远低于2D和1D，揭示了其对高维任务的理解不足。（3）反直觉的是，信息复杂度更高的输入可以简化推理过程。
英文摘要：
Abstract reasoning ability reflects the intelligence and generalization capacity of LLMs to extract and apply abstract rules. However, accurately measuring this ability remains challenging: existing benchmarks either rely on expensive manual annotation, limiting their scale, or risk measuring memorization rather than genuine reasoning. To address this, we introduce an automated pipeline named A2RBench, encompassing generation, expansion, evaluation, and analysis. Specifically, in the generation stage, LLMs create diverse tasks demanding genuine reasoning; in the expansion stage, LLMs reuse validated rules and expand new input spaces to generate task variations, achieving scaling. However, such a process may cause hallucinations. To eliminate it, we further establish a theoretical framework and prove that programmatic verification--testing whether the inverse operation perfectly reverses the forward operation (cycle consistency)--guarantees a unique solution. Through extensive evaluations on mainstream LLMs, we find: (1) Current LLMs exhibit fundamental deficiencies in abstract reasoning, with top models significantly underperforming humans on a representative subset (39.8% vs. 68.5%). (2) Current LLMs fall far short of 2D and 1D in the complexity of generated 3D tasks, revealing their lack of understanding of high-dimensional tasks. (3) Counterintuitively, inputs with higher information complexity can simplify the reasoning process.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
