arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.11997cs.CLcs.AIcs.LG
收藏

记得那年那件事吗?评估大语言模型中的时间信息与时间推理能力

Remember This Event That Year? Assessing Temporal Information and Reasoning in Large Language Models

Indian Institute of Technology Gandhinagar(印度甘地讷格尔理工学院)

Himanshu Beniwal, Dishant Patel, Kowsik Nandagopan D, Hritik Ladia, Ankit Yadav, Mayank Singh

AI总结:

本文通过在时间数据集 TempUN 上实验 12 个大语言模型,提出六项指标评估三种学习范式,揭示模型的时间知识缺口,并证明微调可提升时间推理能力、减少错误输出。

摘要:

大语言模型(LLMs)日益普及,但它们保留和推理时间信息的能力仍然有限,这阻碍了它们在理解事件顺序至关重要的现实场景中的应用。我们的研究在新型数值时间数据集 TempUN 上对 12 个最先进模型(参数量从 2B 到 70B+ 不等)进行了实验,该数据集时间跨度从公元前 10,000 年到公元 2100 年,以揭示显著的时间保留和理解局限。我们提出了六项指标来评估三种学习范式,以增强时间知识获取。我们的研究结果表明,开源模型更频繁地表现出知识缺口,这表明有限知识与错误响应之间存在权衡。此外,各种微调方法显著提高了性能,减少了错误输出,并影响了生成中“信息不可用”的识别。相关数据集和代码可在(https://github.com/lingoiitgn/TempUN)获取。

英文摘要:

Large Language Models (LLMs) are increasingly ubiquitous, yet their ability to retain and reason about temporal information remains limited, hindering their application in real-world scenarios where understanding the sequential nature of events is crucial. Our study experiments with 12 state-of-the-art models (ranging from 2B to 70B+ parameters) on a novel numerical-temporal dataset, \textbf{TempUN}, spanning from 10,000 BCE to 2100 CE, to uncover significant temporal retention and comprehension limitations. We propose six metrics to assess three learning paradigms to enhance temporal knowledge acquisition. Our findings reveal that open-source models exhibit knowledge gaps more frequently, suggesting a trade-off between limited knowledge and incorrect responses. Additionally, various fine-tuning approaches significantly improved performance, reducing incorrect outputs and impacting the identification of 'information not available' in the generations. The associated dataset and code are available at (https://github.com/lingoiitgn/TempUN).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑