谎言，谎言，逻辑泥潭：大语言模型假设推理基准测试
Liar, Liar, Logical Mire: A Benchmark for Suppositional Reasoning in Large Language Models
AI总结：
本文提出基于骑士与无赖逻辑谜题的假设推理基准TruthQuest，评估发现大语言模型在这类任务上存在明显困难，不同性能水平的模型呈现出差异化的推理错误类型。
摘要：
骑士与无赖问题是一类经典的逻辑谜题，其中角色要么说真话，要么说谎。解题目标是根据各角色的陈述，从逻辑上推断出每个角色的身份。这类问题的难点在于说真话或说谎的行为会影响每条陈述的逻辑含义。解决这些谜题不仅需要从单条陈述进行直接推导，还需要具备通过推演多种假设场景来评估陈述真实性的能力。因此，骑士与无赖谜题是假设推理的典型示例。本文中，我们提出了$\textit{TruthQuest}$（真相探索）——一个基于骑士与无赖谜题原理的假设推理基准。该基准包含不同复杂度的问题，同时考量角色数量和涉及的逻辑陈述类型两个维度。在$\textit{TruthQuest}$上的评估显示，Llama 3、Mixtral-8x7B等大语言模型在解决这类任务时存在显著困难。对模型输出的详细错误分析表明，表现较差的模型会出现各类推理错误，且常常无法理解真话与谎言的概念；相比之下，能力更强的模型主要难以准确推断可能为假的陈述的逻辑含义。
英文摘要：
Knights and knaves problems represent a classic genre of logical puzzles where characters either tell the truth or lie. The objective is to logically deduce each character's identity based on their statements. The challenge arises from the truth-telling or lying behavior, which influences the logical implications of each statement. Solving these puzzles requires not only direct deductions from individual statements, but the ability to assess the truthfulness of statements by reasoning through various hypothetical scenarios. As such, knights and knaves puzzles serve as compelling examples of suppositional reasoning. In this paper, we introduce $\textit{TruthQuest}$, a benchmark for suppositional reasoning based on the principles of knights and knaves puzzles. Our benchmark presents problems of varying complexity, considering both the number of characters and the types of logical statements involved. Evaluations on $\textit{TruthQuest}$ show that large language models like Llama 3 and Mixtral-8x7B exhibit significant difficulties solving these tasks. A detailed error analysis of the models' output reveals that lower-performing models exhibit a diverse range of reasoning errors, frequently failing to grasp the concept of truth and lies. In comparison, more proficient models primarily struggle with accurately inferring the logical implications of potentially false statements.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
