基于知识图谱的多跳推理中思维链的直接评估
Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphs
AI总结：
该研究针对大语言模型评估仅关注答案准确率、忽略思维链正确性的问题，提出判别式与生成式结合的思维链评估范式，结合知识图谱探究多跳推理能力，发现模型常通过错误推理得到正确答案。
摘要：
大语言模型（LLMs）在被提示生成伴随答案的思维链（CoT）解释时展现出强大的推理能力。然而，以往评估LLMs的研究仅聚焦于答案准确率，忽略了生成的CoT的正确性。本文中，我们利用知识图谱（KGs）更深入地探究LLMs在多跳问答中的CoT推理能力。我们提出了一种新颖的判别式与生成式结合的CoT评估范式，用以评估LLMs的推理知识以及生成的CoT的准确性。通过在2个多跳问答数据集上对5个不同系列的LLMs开展实验，我们发现LLMs具备足够的知识来执行推理。然而，LLMs的答案准确率与其生成的CoT推理的忠实度之间存在显著差距，这表明它们往往通过错误的推理得出正确答案。
英文摘要：
Large language models (LLMs) demonstrate strong reasoning abilities when prompted to generate chain-of-thought (CoT) explanations alongside answers. However, previous research on evaluating LLMs has solely focused on answer accuracy, neglecting the correctness of the generated CoT. In this paper, we delve deeper into the CoT reasoning capabilities of LLMs in multi-hop question answering by utilizing knowledge graphs (KGs). We propose a novel discriminative and generative CoT evaluation paradigm to assess LLMs' knowledge of reasoning and the accuracy of the generated CoT. Through experiments conducted on 5 different families of LLMs across 2 multi-hop question-answering datasets, we find that LLMs possess sufficient knowledge to perform reasoning. However, there exists a significant disparity between answer accuracy and faithfulness of the CoT reasoning generated by LLMs, indicating that they often arrive at correct answers through incorrect reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
