TiEBe:追踪语言模型对全球重大事件的随时间记忆情况
TiEBe: Tracking Language Model Recall of Notable Worldwide Events Through Time
AI总结:
本研究提出涵盖多地区、多语言、多年份的时事基准TiEBe,评估大语言模型的事件记忆能力,发现其存在地理、语言相关的性能差距,且表现与社会经济指标强相关。
摘要:
随着知识格局不断演变且大语言模型(LLM)日益普及,让这些模型实时更新时事信息的需求愈发迫切。现有基准虽能评估通用事实记忆能力,但鲜有研究探究LLM如何随时间推移或在不同地区间保留知识。为填补这些空白,我们提出了Timely Events Benchmark(TiEBe,时事基准),这是一个包含超23000个问答对的数据集,聚焦全球及地区性重大事件,涵盖10余年的事件、23个地区和13种语言。TiEBe利用来自Wikipedia的结构化回溯数据,识别不同时期的重大事件,随后基于这些事件构建基准,依托Wikipedia之外的事实证据评估LLM对全球及地区发展的理解能力。我们的结果显示,事实记忆存在显著的地理差异,凸显了LLM训练中需要更均衡的全球代表性。我们还观察到,模型在TiEBe上的表现与人类发展指数(HDI)等各国社会经济指标之间的皮尔逊相关系数超过0.7。此外,我们通过以事件发生地区的母语提问,检验了语言对事实记忆的影响,发现低资源语言存在显著的性能差距。
英文摘要:
As the knowledge landscape evolves and large language models (LLMs) become increasingly widespread, there is a growing need to keep these models updated with current events. While existing benchmarks assess general factual recall, few studies explore how LLMs retain knowledge over time or across different regions. To address these gaps, we present the Timely Events Benchmark (TiEBe), a dataset of over 23,000 question-answer pairs centered on notable global and regional events, spanning more than 10 years of events, 23 regions, and 13 languages. TiEBe leverages structured retrospective data from Wikipedia to identify notable events through time. These events are then used to construct a benchmark to evaluate LLMs' understanding of global and regional developments, grounded in factual evidence beyond Wikipedia itself. Our results reveal significant geographic disparities in factual recall, emphasizing the need for more balanced global representation in LLM training. We also observe a Pearson correlation of more than 0.7 between models' performance in TiEBe and various countries' socioeconomic indicators, such as HDI. In addition, we examine the impact of language on factual recall by posing questions in the native language of the region where each event occurred, uncovering substantial performance gaps for low-resource languages.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。