arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.14192cs.CLcs.AI
收藏

Timo:迈向更优的语言模型时间推理能力

Timo: Towards Better Temporal Reasoning for Language Models

Institute of Computer Science and Technology, Soochow University(苏州大学计算机科学与技术学院);Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学)

Zhaochen Su, Jun Zhang, Tong Zhu, Xiaoye Qu, Juntao Li, Min Zhang, Yu Cheng

AI总结:

针对现有大语言模型时间推理方法泛化性不足的问题,研究者结合数学数据增强与自批评时间优化方法,开发了7B和13B规模的Timo模型,在38项时间推理任务上取得同尺寸模型最优性能。

摘要:

时间推理对大语言模型(LLMs)理解世界至关重要。现有研究聚焦于解决特定任务,主要是时间敏感型问答,虽已被证明有效,但无法泛化到更广泛的时间推理任务范畴。因此我们提出核心问题:能否构建一个通用框架来处理各类时间推理任务?为此,我们系统研究了38项时间推理任务,基于其中19项与数学直接相关的观察,首先利用现有数学数据集为时间推理打下坚实基础。但深入研究表明,仅聚焦数学增强不足以解决纯时间推理任务。为缓解这一局限,我们提出一种简单有效的自批评时间优化方法,在不牺牲通用任务能力的前提下增强模型的时间推理能力。最终我们开发了Timo模型,在7B和13B参数规模下均具备出色的时间推理表现。值得注意的是,Timo的平均准确率得分比同级别LLMs分别高出10.0和7.6,在同等规模模型中取得了全新的 state-of-the-art(SOTA,当前最优)性能。大量实验进一步验证了我们框架的有效性及其在各类时间任务上的泛化能力。代码可在https://github.com/zhaochen0110/Timo 获取。

英文摘要:

Reasoning about time is essential for Large Language Models (LLMs) to understand the world. Previous works focus on solving specific tasks, primarily on time-sensitive question answering. While these methods have proven effective, they cannot generalize to a wider spectrum of temporal reasoning tasks. Therefore, we propose a crucial question: Can we build a universal framework to handle a variety of temporal reasoning tasks? To that end, we systematically study 38 temporal reasoning tasks. Based on the observation that 19 tasks are directly related to mathematics, we first leverage the available mathematical dataset to set a solid foundation for temporal reasoning. However, the in-depth study indicates that focusing solely on mathematical enhancement falls short of addressing pure temporal reasoning tasks. To mitigate this limitation, we propose a simple but effective self-critic temporal optimization method to enhance the model's temporal reasoning capabilities without sacrificing general task abilities. Finally, we develop Timo, a model designed to excel in temporal reasoning at the 7B and 13B scales. Notably, Timo outperforms the counterpart LLMs by 10.0 and 7.6 in average accuracy scores and achieves the new state-of-the-art (SOTA) performance of comparable size. Extensive experiments further validate our framework's effectiveness and its generalization across diverse temporal tasks. The code is available at https://github.com/zhaochen0110/Timo.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑