大语言模型缺乏医学知识的时间意识
Large Language Models Lack Temporal Awareness of Medical Knowledge
AI总结：
本文提出TempoMed-Bench基准测试，揭示大语言模型在医学知识时间意识方面的不足，包括知识衰减、历史知识回忆困难及时间不一致行为，指出整合代理搜索工具难以解决该问题。
摘要：
现有评估大语言模型医学知识的方法多基于非时间相关的基准测试，而医学知识本质上是动态变化的。本文构建了首个针对医学领域评估大语言模型时间意识的TempoMed-Bench基准测试。通过分析发现，大语言模型在医学知识的时间意识方面存在显著缺陷：模型对最新医学知识的性能随时间逐渐线性下降，而非出现知识截止行为；模型在回忆过时历史知识时表现较差，准确率仅为最新知识的25.37%-53.89%；且模型在相邻年份间表现出时间不一致的行为。此外，整合代理搜索工具时，时间意识问题难以轻易解决。本文揭示了这一重要但未被充分研究的挑战，推动未来研究发展能更好地编码时间特定医学知识的大语言模型。
英文摘要：
The existing methods for evaluating the medical knowledge of Large Language Models (LLMs) are largely based on atemporal examination-style benchmarks, while in reality, medical knowledge is inherently dynamic and continuously evolves as new evidence emerges and treatments are approved. Consequently, evaluating medical knowledge without a temporal context may provide an incomplete assessment of whether LLMs can accurately reason about time-specific medical knowledge. Moreover, most medical data are historical, requiring the models not only to recall the correct knowledge, but also to know when that knowledge is correct. To bridge the gap, we built TempoMed-Bench, the first-of-its-kind benchmark for evaluating the temporal awareness of the LLMs in the medical domain through evolving guideline knowledge. Based on the TempoMed-Bench, our evaluation analysis first reveals that LLMs lack temporal awareness in medical knowledge through the key findings: (1) model performance on up-to-date medical knowledge exhibits a gradual linear decline over time rather than a sharp knowledge-cutoff behavior, suggesting that parametric medical knowledge is not strictly bounded by knowledge cutoffs; (2) LLMs consistently struggle more with recalling outdated historical medical knowledge than with up-to-date recommendations: accuracy of historical knowledge is only 25.37%-53.89% of up-to-date knowledge, indicating potential knowledge forgetting effects during training; and (3) LLMs often exhibit temporally inconsistent behaviors, where predictions fluctuate irregularly across neighboring years. We also show that the temporal awareness problem is a challenge that cannot be easily solved when integrated with agentic search tools (-3.15%-14.14%). This work highlights an important yet underexplored challenge and motivates future research on developing LLMs that can better encode time-specific medical knowledge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
