大型语言模型是否具有时间基础？
Are Large Language Models Temporally Grounded?
AI总结：
本研究通过三个任务评估LLMs的时间常识，发现其显著落后于人类和专门模型，尤其在自一致性上表现不佳，且模型规模扩大无益，表明LLMs缺乏一致的时间模型。
摘要：
大型语言模型（LLMs）是否具有时间基础？由于LLMs无法感知和与环境互动，直接回答这个问题是不可能的。相反，我们为LLMs提供文本叙述，并针对它们关于事件结构和持续时间的常识知识、沿时间线对事件进行排序的能力以及其时间模型内的自一致性（例如，对于任何一对事件，诸如“之后”和“之前”的时间关系是互斥的）进行探测。我们在反映这些能力的三个任务上评估了最先进的LLMs（如LLaMA 2和GPT-4）。总体而言，我们发现LLMs显著落后于人类表现以及小规模、专门的语言模型。上下文学习、指令微调和思维链提示仅在一定程度上缩小了这一差距。关键的是，LLMs在自一致性方面表现最差，在其至少27.23%的预测中显示出不一致的行为。与预期相反，我们还发现扩大模型规模并不能保证性能的积极提升。为了解释这些结果，我们研究了LLMs可能获取时间信息的来源：我们发现，在预训练期间可用的未标记文本中的句子排序与事件排序仅弱相关。此外，公开的指令微调混合包含很少的时间任务。因此，我们得出结论，当前的LLMs缺乏对文本叙述的一致时间模型。代码、数据集和LLM输出可在https://github.com/yfqiu-nlp/temporal-llms获取。
英文摘要：
Are Large language models (LLMs) temporally grounded? Since LLMs cannot perceive and interact with the environment, it is impossible to answer this question directly. Instead, we provide LLMs with textual narratives and probe them with respect to their common-sense knowledge of the structure and duration of events, their ability to order events along a timeline, and self-consistency within their temporal model (e.g., temporal relations such as after and before are mutually exclusive for any pair of events). We evaluate state-of-the-art LLMs (such as LLaMA 2 and GPT-4) on three tasks reflecting these abilities. Generally, we find that LLMs lag significantly behind both human performance as well as small-scale, specialised LMs. In-context learning, instruction tuning, and chain-of-thought prompting reduce this gap only to a limited degree. Crucially, LLMs struggle the most with self-consistency, displaying incoherent behaviour in at least 27.23% of their predictions. Contrary to expectations, we also find that scaling the model size does not guarantee positive gains in performance. To explain these results, we study the sources from which LLMs may gather temporal information: we find that sentence ordering in unlabelled texts, available during pre-training, is only weakly correlated with event ordering. Moreover, public instruction tuning mixtures contain few temporal tasks. Hence, we conclude that current LLMs lack a consistent temporal model of textual narratives. Code, datasets, and LLM outputs are available at https://github.com/yfqiu-nlp/temporal-llms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
