思维链的强度取决于其最薄弱的环节：推理链验证器基准测试
A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains
AI总结：
针对缺乏细粒度步骤级数据集评估推理验证方法的问题，研究提出REVEAL基准数据集，用于开放域问答下思维链推理验证器测试，标注涵盖步骤相关性、证据归因与逻辑正确性，实验发现验证器在逻辑正确性验证和矛盾检测上表现不佳。
摘要：
提示语言模型提供逐步答案（例如“思维链（Chain-of-Thought）”）是复杂推理任务的主流方法，通常更准确的推理链会提升下游任务性能。近期文献探讨了自动验证推理的方法，以评估并提高其正确性。然而，目前尚无细粒度的步骤级数据集可对这类验证方法进行全面评估，阻碍了该方向的进展。我们推出REVEAL（推理验证评估，Reasoning Verification Evaluation）数据集，用于在开放域问答场景下对复杂思维链推理的自动验证器进行基准测试。REVEAL包含跨多个数据集和前沿语言模型的全面标注，涵盖语言模型答案中每个推理步骤的相关性、对证据段落的归因以及逻辑正确性。在REVEAL上的评估显示，验证器在验证推理链方面存在困难，尤其是在验证逻辑正确性和检测矛盾方面。数据集可访问https://reveal-dataset.github.io/ 获取。
英文摘要：
Prompting language models to provide step-by-step answers (e.g., "Chain-of-Thought") is the prominent approach for complex reasoning tasks, where more accurate reasoning chains typically improve downstream task performance. Recent literature discusses automatic methods to verify reasoning to evaluate and improve their correctness. However, no fine-grained step-level datasets are available to enable thorough evaluation of such verification methods, hindering progress in this direction. We introduce REVEAL: Reasoning Verification Evaluation, a dataset to benchmark automatic verifiers of complex Chain-of-Thought reasoning in open-domain question-answering settings. REVEAL includes comprehensive labels for the relevance, attribution to evidence passages, and logical correctness of each reasoning step in a language model's answer, across a variety of datasets and state-of-the-art language models. Evaluation on REVEAL shows that verifiers struggle at verifying reasoning chains - in particular, verifying logical correctness and detecting contradictions. Available at https://reveal-dataset.github.io/ .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
