自然语言证明规划中的演绎可加性
Deductive Additivity for Planning of Natural Language Proofs
AI总结：
本文研究自然语言多步推理中能否利用嵌入空间的演绎可加性实现高效规划，通过评估多种稠密与稀疏嵌入模型并构建 SSRC 数据集，发现现有嵌入方法虽常将结论嵌入到前提之和附近，但仍不足以作为有效启发式。
摘要：
当前用于多步声明验证的自然语言系统通常分两个阶段运行：使用启发式方法检索一组相关前提陈述（规划），然后使用大型语言模型从这些陈述中生成新结论（演绎）。规划步骤通常需要昂贵的 Transformer 运算，并且无法扩展到任意数量的前提陈述。在本文中，我们研究是否可以通过与演绎推理兼容的嵌入空间实现一种高效的规划启发式方法。具体而言，我们评估嵌入空间是否表现出一种我们称之为演绎可加性的性质：前提陈述嵌入的和应当接近基于这些前提所得结论的嵌入。除了来自 GPT3 的微调嵌入和来自 BM25 的稀疏嵌入之外，我们还探索了多种现成的稠密嵌入来源。我们从内在和外在两个角度研究嵌入模型：内在评估演绎可加性这一性质是否成立，外在则利用它们辅助自然语言证明生成中的规划。最后，我们创建了一个数据集 Single-Step Reasoning Contrast (SSRC)，以进一步探究模型在各种推理类型上的表现。我们的研究结果表明，尽管标准嵌入方法经常将结论嵌入到其前提嵌入之和的附近，但它们仍不足以成为有效的启发式方法，并且缺乏对某些推理类别进行建模的能力。
英文摘要：
Current natural language systems designed for multi-step claim validation typically operate in two phases: retrieve a set of relevant premise statements using heuristics (planning), then generate novel conclusions from those statements using a large language model (deduction). The planning step often requires expensive Transformer operations and does not scale to arbitrary numbers of premise statements. In this paper, we investigate whether an efficient planning heuristic is possible via embedding spaces compatible with deductive reasoning. Specifically, we evaluate whether embedding spaces exhibit a property we call deductive additivity: the sum of premise statement embeddings should be close to embeddings of conclusions based on those premises. We explore multiple sources of off-the-shelf dense embeddings in addition to fine-tuned embeddings from GPT3 and sparse embeddings from BM25. We study embedding models both intrinsically, evaluating whether the property of deductive additivity holds, and extrinsically, using them to assist planning in natural language proof generation. Lastly, we create a dataset, Single-Step Reasoning Contrast (SSRC), to further probe performance on various reasoning types. Our findings suggest that while standard embedding methods frequently embed conclusions near the sums of their premises, they fall short of being effective heuristics and lack the ability to model certain categories of reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
