LGMT：基于逻辑的蜕变测试用于评估LLMs的推理可靠性
LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs
AI总结：
提出LGMT框架，利用一阶逻辑推导蜕变关系，通过一致性检查评估LLM推理的鲁棒性，揭示传统评估忽略的隐藏缺陷。
摘要：
大型语言模型（LLMs）在逻辑推理基准测试中表现出色，但其可靠性仍不确定。现有评估依赖静态基准，无法评估在逻辑等价变换下的鲁棒性，且往往高估推理能力。我们提出LGMT（基于逻辑的蜕变测试），一种无神谕框架，利用一阶逻辑（FOL）评估LLM推理。通过从形式逻辑等价推导蜕变关系，LGMT构建语义不变的测试用例，并通过跨案例一致性检查检测推理缺陷。在六个最先进的LLM上的实验表明，LGMT暴露了传统基于参考的评估遗漏的大量隐藏缺陷。我们进一步发现，模型对符号级别和结论级别的变化特别敏感，而高级提示如Few-shot CoT仅能部分缓解这些问题。这些结果表明，LLM评估应从孤立的正确性转向逻辑不变性下的鲁棒性。LGMT为诊断推理失败提供了一种原则性和可扩展的方法。
英文摘要：
Large Language Models (LLMs) achieve strong performance on logical reasoning benchmarks, yet their reliability remains uncertain. Existing evaluations rely on static benchmarks, which fail to assess robustness under logically equivalent transformations and often overestimate reasoning capability. We propose LGMT (Logic-Grounded Metamorphic Testing), an oracle-free framework that leverages first-order logic (FOL) to evaluate LLM reasoning. By deriving metamorphic relations from formal logical equivalences, LGMT constructs semantically invariant test cases and detects reasoning defects through cross-case consistency checking. Experiments on six state-of-the-art LLMs show that LGMT exposes substantial hidden defects missed by traditional reference-based evaluations. We further find that models are particularly sensitive to symbol-level and conclusion-level variations, and that advanced prompting such as Few-shot CoT only partially mitigates these issues. These results suggest that LLM evaluation should move beyond isolated correctness toward robustness under logical invariance. LGMT provides a principled and scalable approach for diagnosing reasoning failures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
