大语言模型中逻辑一致性的可控重新表述测试
Controlled Reformulation Testing for Logical Consistency in Large Language Models
- University of Texas at Austin（德克萨斯大学奥斯汀分校）
机构由 AI 辅助整理，请以论文原文为准。
中文总结 AI 辅助
研究大语言模型在逻辑等价问题表面形式改变时的逻辑一致性，提出CRTBench基准，评估多个前沿LLMs，发现存在准确率 - 一致性差距，揭示失败集中在逻辑非平凡变换，表明仅靠准确率无法评估LLMs的逻辑推理。
中文摘要 AI 辅助
大语言模型（LLMs）在逻辑等价问题的表面形式改变时常常自相矛盾。我们提出了一个用于可控重新表述测试（CRTBench）的基准，包含350个问题族（共1750个问题）来评估逻辑不变性。在此基准中，研究LLMs在包括逆否改写、双重否定、否定翻转和被动语态等可控重新表述下保持一致答案的能力。评估了几个前沿LLMs，发现存在准确率 - 一致性差距，如GPT - 5.4 - mini基础准确率达98.9%，但族级一致性仅60.3%，而推理优化的o4 - mini一致性达96.9%。实验表明，失败集中在逻辑非平凡变换如逆否改写（GPT - 5.4 - mini为72.4%）和双重否定（84.6%），而表面级改写较稳健（94 - 100%）。增加推理努力可将GPT - 5.4 - mini一致性提高到85.4%，但整体上GPT - 5.4无变化，因为量词族上的失败抵消了嵌套否定上的收益。这些结果表明仅靠准确率不足以评估LLMs中的逻辑推理。
英文摘要
Large language models (LLMs) frequently contradict themselves when the surface form of a logically equivalent question changes. We present a benchmark of 350 question families (1,750 total questions) for Controlled Reformulation Testing (CRTBench) to evaluate logical invariance. In this benchmark, we investigate LLMs' ability to maintain consistent answers across controlled reformulations, which include contrapositive rewriting, double negation, negation flipping, and passive voice. We evaluate several frontier LLMs and observe an accuracy-consistency gap where GPT-5.4-mini achieves $98.9\%$ base accuracy but only $60.3\%$ family-level consistency, while reasoning-optimized o4-mini achieves $96.9\%$ consistency. From our experiments, we observe that failures cluster around logically nontrivial transformations such as contrapositive rewriting ($72.4\%$ for GPT-5.4-mini) and double negation ($84.6\%$), while surface-level rephrasing remains robust ($94-100\%$). Increasing reasoning effort improves GPT-5.4-mini to $85.4\%$ consistency, but leaves GPT-5.4 unchanged overall because gains on nested negation are offset by failures on quantifier families. These results show that accuracy alone is not enough for evaluating logical reasoning in LLMs.
