幻影回忆：当熟悉的谜题迷惑智能模型
PHANTOM RECALL: When Familiar Puzzles Fool Smart Models
AI总结：
本研究提出PHANTOM RECALL基准，通过扰动经典逻辑谜题，揭示大型语言模型依赖记忆而非推理的脆弱性，并提供判断器、错误分类和提示缓解框架，表明模型在情境变化时缺乏重新推理能力。
摘要：
大型语言模型（LLMs），如GPT、Gemini和Claude，通常看起来擅长解决经典逻辑谜题——但它们的答案背后有多少真正的推理呢？最近的证据表明，这些模型经常依赖记忆的模板，而非从第一性原理进行推理。当谜题被稍作修改时，它们的性能就会崩溃，暴露出惊人的脆弱性。特别是，我们提出疑问：LLMs是否解决了这些问题？解决到什么程度？对其他谜题的扰动又如何？是否存在一种通用的提示重述方式，使模型表现更好？为了系统地研究这些，我们引入了PHANTOM RECALL，一个包含25个著名逻辑谜题和149个精心设计的扰动的基准，这些扰动保留了推理结构，但改变了表面细节和解决方案。我们评估了十一个领先的LLMs，并识别出一个反复出现的失败模式——幻影回忆——即模型自信地重现记忆的解决方案或虚假的推理，而这些不再适用于改变后的场景。为了探究和缓解这个问题，我们贡献了三个工具：（i）一个自动逻辑等价判断器，用于检测推理不匹配；（ii）一个细粒度推理错误分类法；以及（iii）一个由这些类别指导的基于提示的缓解框架。尽管在未修改的谜题上准确率接近完美，但模型在扰动谜题上的表现显著不如人类，表现出幻影回忆和过度阐述。我们的发现揭示了一个关键限制：当上下文线索变化时，LLMs常常无法重新推理——凸显了语言流畅性与逻辑理解之间的差距。
英文摘要：
Large language models (LLMs) such as GPT, Gemini, and Claude often appear adept at solving classic logic puzzles--but how much genuine reasoning underlies their answers? Recent evidence suggests that these models frequently rely on memorized templates rather than reasoning from first principles. When puzzles are slightly modified, their performance collapses, revealing a striking fragility. In particular, we asked: Have LLMs addressed these issues? To what extent? How about perturbations to other puzzles? Is there a general way of reformulating the prompt so that the models do better? To examine these things systematically, we introduce PHANTOM RECALL, a benchmark comprising 25 well-known logic puzzles and 149 carefully designed perturbations that preserve reasoning structure but alter superficial details and solutions. We evaluate eleven leading LLMs and identify a recurring failure mode--phantom recall--where models confidently reproduce memorized solutions or spurious rationales that no longer fit the altered scenario. To probe and mitigate this issue, we contribute three tools: (i) an automated logical-equivalence judge to detect reasoning mismatches, (ii) a taxonomy of fine-grained reasoning error categories, and (iii) a prompting-based mitigation framework guided by these categories. Despite near-perfect accuracy on unmodified puzzles, models significantly underperform humans on perturbed ones, exhibiting both phantom recall and over-elaboration. Our findings reveal a crucial limitation: LLMs often fail to re-reason when contextual cues shift--highlighting the gap between linguistic fluency and logical understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
