arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.01290cs.CLcs.AI
收藏

Knowledge Crosswords:基于大语言模型的几何知识推理

Knowledge Crosswords: Geometric Knowledge Reasoning with Large Language Models

The Hong Kong University of Science and Technology(香港科技大学);University of Washington(华盛顿大学);Xi’an Jiaotong University(西安交通大学);University of Notre Dame(圣母大学);Carnegie Mellon University(卡内基梅隆大学)

Wenxuan Ding, Shangbin Feng, Yuhan Liu, Zhaoxuan Tan, Vidhisha Balachandran, Tianxing He, Yulia Tsvetkov

AI总结:

该研究提出几何知识推理基准Knowledge Crosswords,针对现有LLM在该任务上的不足,提出分阶段提示和全量验证方法,其中全量验证方法性能更优且鲁棒性更强,揭示了几何知识推理对LLM的新挑战。

摘要:

我们提出Knowledge Crosswords(知识填字游戏),这是一个几何知识推理基准,由受结构化事实约束限定的不完整知识网络构成,任务是让大语言模型(LLM)推断缺失的事实以满足所有约束。这种几何知识推理的全新设定要求语言模型具备现有原子/线性多跳问答之外的新能力,例如回溯、事实与约束验证、不确定性推理等。Knowledge Crosswords包含2101个独立问题,覆盖多样知识领域,并进一步划分为三个难度等级。我们开展了大量实验,在该基准上评估现有LLM及相关方法。结果表明,基线方法在处理更大的知识网络和语义等价的实体干扰项时表现不佳。针对这些局限,我们提出两种新方法:分阶段提示(Staged Prompting)和全量验证(Verify-All),以增强LLM的错误感知回溯和约束验证能力。我们的Verify-All方法显著优于先前方法,且对困难子集的问题鲁棒性更强。进一步分析显示,几何知识推理对LLM的知识能力提出了新挑战,尤其是在对选项顺序变化的鲁棒性、知识网络中的复杂结构约束、“以上都不对”场景等方面。

英文摘要:

We propose Knowledge Crosswords, a geometric knowledge reasoning benchmark consisting of incomplete knowledge networks bounded by structured factual constraints, where LLMs are tasked with inferring the missing facts to meet all constraints. The novel setting of geometric knowledge reasoning necessitates new LM abilities beyond existing atomic/linear multi-hop QA, such as backtracking, verifying facts and constraints, reasoning with uncertainty, and more. Knowledge Crosswords contains 2,101 individual problems, covering diverse knowledge domains, and is further divided into three difficulty levels. We conduct extensive experiments to evaluate existing LLMs and approaches on Knowledge Crosswords. Results demonstrate that baseline approaches struggle with larger knowledge networks and semantically-equivalent entity distractors. In light of their limitations, we propose two new approaches, Staged Prompting and Verify-All, to augment LLMs' abilities for error-aware backtracking and constraint verification. Our Verify-All significantly outperforms prior methods and is more robust towards problems in the hard subset. Further analysis shows that geometric knowledge reasoning poses new challenges to LLMs' knowledge abilities, particularly in robustness towards varying option orders, complex structural constraints in knowledge networks, "none of the above" scenarios, and more.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑