通过符号验证评估逐步推理
Evaluating Step-by-Step Reasoning through Symbolic Verification
AI总结：
本文构建自然语言与符号等价数据集，提出神经符号方法LMLP，通过逻辑规则迭代推理并支持自动验证，在演绎推理中比思维链准确率提升超过25%。
摘要：
预训练语言模型（LMs）在使用解释或思维链（CoT）进行上下文学习时，已展现出显著的推理性能。另一方面，这些推理任务通常被认为更适合用符号编程来处理。为了理解语言模型的推理机制，我们构建了包含等价（自然语言、符号）数据对的合成数据集，其中符号示例包含来自非参数知识库（KBs）的一阶逻辑规则和谓词，支持对中间推理结果进行自动验证。随后，我们重新审视神经符号方法，并提出从包含逻辑规则和对应示例的演示中学习，以在知识库上迭代推理，恢复Prolog的后向链算法，并支持对语言模型输出的自动验证。我们进行了全面的实验，在演绎推理设置中系统比较LMLP与CoT，结果表明LMLP在长度泛化基准上比CoT的准确率高出超过25%，即使模型规模更小。
英文摘要：
Pre-trained language models (LMs) have shown remarkable reasoning performance using explanations or chain-of-thoughts (CoT)) for in-context learning. On the other hand, these reasoning tasks are usually presumed to be more approachable for symbolic programming. To understand the mechanism of reasoning of LMs, we curate synthetic datasets containing equivalent (natural, symbolic) data pairs, where symbolic examples contain first-order logic rules and predicates from non-parametric knowledge bases (KBs), supporting automated verification of intermediate reasoning results. Then we revisit neuro-symbolic approaches and propose to learn from demonstrations containing logic rules and corresponding examples to iteratively reason over KBs, recovering Prolog's backward chaining algorithm and supporting automated verification of LMs' outputs. Comprehensive experiments are included to systematically compare LMLP with CoT in deductive reasoning settings, showing that LMLP enjoys more than $25\%$ higher accuracy than CoT on length generalization benchmarks even with smaller model sizes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
