arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.09170cs.CL
收藏

时间考验:评估大语言模型时间推理能力的基准

Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning

Google Research(谷歌研究院);Google DeepMind(谷歌DeepMind);Google(谷歌公司)

Bahare Fatemi, Mehran Kazemi, Anton Tsitsulin, Karishma Malkan, Jinyeong Yim, John Palowitch, Sungyong Seo, Jonathan Halcrow, Bryan Perozzi

AI总结:

本文提出合成基准Test of Time,通过多样化问题类型系统评估LLM的时间推理能力,揭示其优势与不足,并开源数据集与评估框架。

摘要:

大语言模型(LLMs)展示了卓越的推理能力,但在涉及复杂时间逻辑的时间推理任务中仍然容易出错。现有研究利用多样化的数据集和基准探索了LLM在时间推理上的表现。然而,这些研究通常依赖LLM可能在预训练期间接触过的真实世界数据,或采用可能无意中引入事实不一致性的匿名化技术。在本工作中,我们通过引入专门设计用于评估LLM在多种场景下时间推理能力的新型合成数据集来解决这些局限性。这些数据集中问题类型的多样性使得能够系统性地研究问题结构、规模、问题类型、事实顺序及其他因素对LLM性能的影响。我们的发现为当前LLM在时间推理任务中的优势与不足提供了有价值的见解。为促进该领域的进一步研究,我们开源了实验中使用的数据集和评估框架:https://huggingface.co/datasets/baharef/ToT。

英文摘要:

Large language models (LLMs) have showcased remarkable reasoning capabilities, yet they remain susceptible to errors, particularly in temporal reasoning tasks involving complex temporal logic. Existing research has explored LLM performance on temporal reasoning using diverse datasets and benchmarks. However, these studies often rely on real-world data that LLMs may have encountered during pre-training or employ anonymization techniques that can inadvertently introduce factual inconsistencies. In this work, we address these limitations by introducing novel synthetic datasets specifically designed to assess LLM temporal reasoning abilities in various scenarios. The diversity of question types across these datasets enables systematic investigation into the impact of the problem structure, size, question type, fact order, and other factors on LLM performance. Our findings provide valuable insights into the strengths and weaknesses of current LLMs in temporal reasoning tasks. To foster further research in this area, we are open-sourcing the datasets and evaluation framework used in our experiments: https://huggingface.co/datasets/baharef/ToT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑