深入探讨大型语言模型在逻辑推理中的自我验证能力
A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning
AI总结：
本文在逻辑推理背景下深入探讨了LLMs的自我验证能力，通过引入包含232种推理谬误的FALLACIES数据集进行实验，发现现有模型难以准确识别谬误步骤，无法保证自我验证有效性，并据此提出了未来研究建议。
摘要：
逻辑推理一直是人工智能领域持续追求的目标。尽管大型语言模型（LLMs）取得了显著进展，但它们在处理复杂的逻辑推理问题时仍然存在困难。为了提升推理性能，一个极具前景的方向是可扩展监督，这要求LLMs识别自身的错误并随后进行自我改进。为了实现这一目标，人们提出了各种自我验证方法。然而，现有模型是否能充分理解自身的错误仍处于研究之中。本文在逻辑推理的背景下，深入探讨了LLMs的自我验证能力，重点关注其准确识别逻辑谬误的能力。我们引入了一个名为FALLACIES的数据集，其中包含232种按层次分类法进行分类的推理谬误。通过对FALLACIES进行详尽的实验，我们对一系列模型的验证能力进行了全面且详细的分析。我们的主要发现表明，现有的LLMs可能难以准确识别谬误推理步骤，并且可能无法保证自我验证方法的有效性。基于这些观察结果，我们为自我验证方法的未来研究和实际应用提供了建议。
英文摘要：
Logical reasoning has been an ongoing pursuit in the field of AI. Despite significant advancements made by large language models (LLMs), they still struggle with complex logical reasoning problems. To enhance reasoning performance, one promising direction is scalable oversight, which requires LLMs to identify their own errors and then improve by themselves. Various self-verification methods have been proposed in pursuit of this goal. Nevertheless, whether existing models understand their own errors well is still under investigation. In this paper, we take a closer look at the self-verification abilities of LLMs in the context of logical reasoning, focusing on their ability to identify logical fallacies accurately. We introduce a dataset, FALLACIES, containing 232 types of reasoning fallacies categorized in a hierarchical taxonomy. By conducting exhaustive experiments on FALLACIES, we obtain comprehensive and detailed analyses of a series of models on their verification abilities. Our main findings suggest that existing LLMs could struggle to identify fallacious reasoning steps accurately and may fall short of guaranteeing the validity of self-verification methods. Drawing from these observations, we offer suggestions for future research and practical applications of self-verification methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
