ChroKnowledge：揭示语言模型在多个领域中的时间顺序知识
ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domains
AI总结：
针对LLM时间顺序知识评估难题，提出基准ChroKnowBench与框架ChroKnowledge及提示方法ChroKnowPrompt，实验显示其能有效激发多领域时间知识。
摘要：
大型语言模型（LLMs）给我们的生活许多方面带来了重大变化。然而，评估和确保它们的时间顺序知识仍然具有挑战性。现有方法在解决知识的时域适应性方面存在不足，往往依赖于固定的时间点视角。为了克服这一点，我们引入了ChroKnowBench，一个基准数据集，旨在从三个关键方面评估按时间顺序累积的知识：多领域、时间依赖性和时间状态。我们的基准区分了会演变的知识（例如，个人历史、科学发现、修订的法律）和保持不变的知识（例如，数学真理、常识性事实）。基于这一基准，我们提出了ChroKnowledge（知识的时间顺序分类），一个新颖的基于采样的框架，用于评估LLMs的非参数化时间顺序知识。我们的评估得出以下观察结果：（1）激发时间知识的能力因模型训练所依据的数据格式而异。（2）LLMs部分地回忆知识，或在时间边界处表现出截断，而不是正确地回忆知识的所有方面。因此，我们应用了我们的ChroKnowPrompt，一种深入的提示方法，通过逐步遍历周围的时间跨度来激发时间顺序知识。我们观察到，它成功地回忆了开源和专有LLMs中的对象，展示了其通用性，尽管它在处理动态数据集和非结构化格式时面临挑战。
英文摘要：
Large language models (LLMs) have brought significant changes to many aspects of our lives. However, assessing and ensuring their chronological knowledge remains challenging. Existing approaches fall short in addressing the temporal adaptability of knowledge, often relying on a fixed time-point view. To overcome this, we introduce ChroKnowBench, a benchmark dataset designed to evaluate chronologically accumulated knowledge across three key aspects: multiple domains, time dependency, temporal state. Our benchmark distinguishes between knowledge that evolves (e.g., personal history, scientific discoveries, amended laws) and knowledge that remain constant (e.g., mathematical truths, commonsense facts). Building on this benchmark, we present ChroKnowledge (Chronological Categorization of Knowledge), a novel sampling-based framework for evaluating LLMs' non-parametric chronological knowledge. Our evaluation led to the following observations: (1) The ability of eliciting temporal knowledge varies depending on the data format that model was trained on. (2) LLMs partially recall knowledge or show a cut-off at temporal boundaries rather than recalling all aspects of knowledge correctly. Thus, we apply our ChroKnowPrompt, an in-depth prompting to elicit chronological knowledge by traversing step-by-step through the surrounding time spans. We observe that it successfully recalls objects across both open-source and proprietary LLMs, demonstrating versatility, though it faces challenges with dynamic datasets and unstructured formats.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
