arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.09072cs.CL
收藏

活在当下：大型语言模型能否掌握共时推理？

Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?

Institute of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）；Shanghai AI Laboratory（上海人工智能实验室）；Huazhong University of Science and Technology（华中科技大学）；Tsinghua University（清华大学）；The Chinese University of Hong Kong（香港中文大学）

Zhaochen Su, Juntao Li, Jun Zhang, Tong Zhu, Xiaoye Qu, Pan Zhou, Yan Bowen, Yu Cheng, Min zhang

备注：
This paper has been accepted to the ACL 2024 main conference
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出共时问答基准 CoTempQA，包含四种共时场景和4,748个样本，发现现有LLMs与人类水平存在显著差距，并提出从数学角度提升共时推理能力的策略。

摘要：

时间推理是大型语言模型（LLMs）理解世界的基础。当前的时间推理数据集仅限于关于单一或孤立事件的问题，未能反映具有并发性和复杂时间关联的现实时间特征。在本文中，我们提出了 CoTempQA，一个全面的共时问答（QA）基准，包含四种共时场景（Equal、Overlap、During、Mix），共 4,748 个样本，用于评估 LLMs 的共时理解和推理能力。我们的大量实验揭示了当前 LLMs 在 CoTempQA 任务上的表现与人类水平推理之间存在显著差距。即使结合了思维链（CoT）方法，模型在我们的任务上仍然持续表现困难。在我们的初步探索中，我们发现数学推理在处理共时事件中发挥着重要作用，并提出了一种从数学角度提升 LLMs 共时推理能力的策略。我们希望我们的 CoTempQA 数据集能够推动进一步提升 LLMs 共时推理能力的进展。我们的代码可在 https://github.com/zhaochen0110/Cotempqa 获取。

英文摘要：

Temporal reasoning is fundamental for large language models (LLMs) to comprehend the world. Current temporal reasoning datasets are limited to questions about single or isolated events, falling short in mirroring the realistic temporal characteristics involving concurrent nature and intricate temporal interconnections. In this paper, we introduce CoTempQA, a comprehensive co-temporal Question Answering (QA) benchmark containing four co-temporal scenarios (Equal, Overlap, During, Mix) with 4,748 samples for evaluating the co-temporal comprehension and reasoning abilities of LLMs. Our extensive experiments reveal a significant gap between the performance of current LLMs and human-level reasoning on CoTempQA tasks. Even when enhanced with Chain of Thought (CoT) methodologies, models consistently struggle with our task. In our preliminary exploration, we discovered that mathematical reasoning plays a significant role in handling co-temporal events and proposed a strategy to boost LLMs' co-temporal reasoning from a mathematical perspective. We hope that our CoTempQA datasets will encourage further advancements in improving the co-temporal reasoning capabilities of LLMs. Our code is available at https://github.com/zhaochen0110/Cotempqa.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑