评估一阶逻辑（FOL）贴近度指标的敏感性与对齐性
Assessing the Sensitivity and Alignment of FOL Closeness Metrics
AI总结：
针对FOL正确性缺乏可靠评估指标的问题，该研究测试了多类FOL贴近度指标的敏感性及与LLM评判的对齐性，发现BertScore对齐度更高，组合指标可同时提升鲁棒性与敏感性。
摘要：
近期借助工具增强的大语言模型（LLM）求解逻辑推理问题的成功范式，利用了将自然语言（NL）语句转换为一阶逻辑（First-Order Logic，FOL）并调用外部定理证明器的思路。然而，由于缺乏可靠的评估指标来比较生成的FOL与真值FOL，包含运算符与文本的FOL语句的正确性往往得不到验证。本文针对现有基于自然语言、FOL和图的指标在捕捉采样FOL与其对应真值之间差异方面的敏感性开展了全面研究，随后衡量了基于指标的FOL输出排序与作为评判者的强性能LLM之间的对齐程度。为此，我们首先对真值FOL语句施加基于运算符和文本的扰动，以评估指标的敏感性；接着通过将指标与LLM判断结果对比，评估指标的鲁棒性。实证结果表明，n-gram指标BLEU对文本扰动存在明显的过度敏感问题；运算符扰动会影响语义图指标Smatch++对结构变化的判断，以及FOL指标对特定运算符变化的判断。我们观察到BertScore与LLM判断的对齐程度更高，证明了语义评估的重要性。此外，研究还表明，与使用单一指标相比，组合指标能同时提升鲁棒性与敏感性。
英文摘要：
The recent successful paradigm of solving logical reasoning problems with tool-augmented large language models (LLMs) leverages translation of natural language (NL) statements into First-Order Logic~(FOL) and external theorem provers. However, the correctness of FOL statements, comprising operators and text, often go unverified due to the lack of a reliable evaluation metric for comparing generated and ground-truth FOLs. In this paper, we conduct a comprehensive study on the sensitivity of existing NL-, FOL-, and graph-based metrics to capture differences between a sampled FOL and its corresponding ground-truth. We then measure the alignment between a metric-based ranking of FOL outputs and a strong LLM as-a-judge. To do this, we first apply operator and text-based perturbations to ground-truth FOL statements to assess metric sensitivity. We then evaluate metric robustness by comparing the metrics against LLMs judgment. Our empirical findings highlight a clear oversensitivity in the n-gram metric BLEU for text perturbations. The operator perturbation affects the semantic graph metric Smatch++ for structural changes, and the FOL metric for specific operator changes. We observe a closer alignment between BertScore and LLM judgement, proving the importance of semantic evaluation. Additionally, we show that combining metrics enhances both robustness and sensitivity compared to using individual metrics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
