arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.09158cs.AI
收藏

改进大型语言模型在事件关系逻辑预测中的表现

Improving Large Language Models in Event Relation Logical Prediction

Peking University(北京大学);Microsoft Research Asia(微软亚洲研究院);Nanyang Technological University(南洋理工大学);School of Computer Science, Fudan University(复旦大学计算机科学学院)

Meiqi Chen, Yubo Ma, Kaitao Song, Yixin Cao, Yan Zhang, Dongsheng Li

AI总结:

本文针对LLM事件关系逻辑不一致问题,比较三种逻辑增强方法,并构建含高阶推理的LLM-ERL数据集用于评估与微调,验证了方法有效性。

摘要:

事件关系对于叙事理解和推理至关重要。事件关系抽取(ERE)受微妙逻辑支配,是一项具有挑战性的任务,需要透彻的语义理解和严谨的逻辑推理。本文进行了深入研究,以系统地探究LLM在理解和应用事件关系逻辑方面的能力。具体而言,我们首先考察了LLM在不同任务中逻辑推理的不足。研究表明,LLM并非逻辑一致的推理者,这导致其在需要严谨推理的任务上表现欠佳。为解决该问题,我们探索了三种不同方法,为LLM赋予事件关系逻辑,从而使其能够在各种场景中生成更连贯的答案。基于所提方法,我们还贡献了一个包含高阶推理的合成数据集LLM-ERL,用于评估和微调。在不同任务上开展的大量定量与定性分析也验证了这些方法的有效性,并为未来利用LLM解决实际任务提供了洞见。代码位于 https://github.com/chenmeiqii/Teach-LLM-LR。

英文摘要:

Event relations are crucial for narrative understanding and reasoning. Governed by nuanced logic, event relation extraction (ERE) is a challenging task that demands thorough semantic understanding and rigorous logical reasoning. In this paper, we conduct an in-depth investigation to systematically explore the capability of LLMs in understanding and applying event relation logic. More in detail, we first investigate the deficiencies of LLMs in logical reasoning across different tasks. Our study reveals that LLMs are not logically consistent reasoners, which results in their suboptimal performance on tasks that need rigorous reasoning. To address this, we explore three different approaches to endow LLMs with event relation logic, and thus enable them to generate more coherent answers across various scenarios. Based on our approach, we also contribute a synthesized dataset (LLM-ERL) involving high-order reasoning for evaluation and fine-tuning. Extensive quantitative and qualitative analyses on different tasks also validate the effectiveness of our approaches and provide insights for solving practical tasks with LLMs in future work. Codes are available at https://github.com/chenmeiqii/Teach-LLM-LR.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑