LibEvoBench：探测代码生成模型中的时间知识分层
LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models
AI总结：
针对LLM在代码生成中因训练数据时间混合导致API版本混淆的问题，提出多版本基准LibEvoBench和新指标SEUS，揭示模型对版本不敏感且仅靠文档可提升准确性。
摘要：
大型软件项目通常依赖旧版本的库，即使API在不同版本中持续演化。这给LLM带来了挑战：它们必须维护多个API版本的知识，而不仅仅是最新或最常见的版本。然而，当前的LLM在时间混合的语料库上训练，缺乏针对此类版本特定推理的显式机制，导致时代错误——调用不同库版本中的API。为了系统评估这一现象，我们引入了LibEvoBench，这是一个涵盖广泛使用的Python库多个版本的多任务基准，以及一个新的度量标准——软件演化理解分数（SEUS），用于衡量模型在处理演化API时的一致性。我们的结果表明，最先进的模型在很大程度上对版本不敏感：对于演化API，性能下降，而对于稳定API，不同版本间性能保持不变。此外，简单地指定目标版本没有带来任何好处，而相关文档显著提高了模型的准确性。这些发现突显了当前训练范式的系统性局限，并激发了代码生成中基于时间知识的新方法。
英文摘要：
Large software projects often depend on older versions of libraries, even as APIs continue to evolve across releases. This creates a challenge for LLMs: they must maintain knowledge of multiple API versions, not merely the latest or most common one. However, current LLMs are trained on temporally mixed corpora and lack explicit mechanisms for such version-specific reasoning, leading to anachronistic errors - calling APIs as they exist in a different library version. To systematically evaluate this phenomenon, we introduce LibEvoBench, a multi-task benchmark spanning multiple versions of widely used Python libraries, along with a new metric, the Software Evolution Understanding Score (SEUS), to measure models' consistency when working with evolving APIs. Our results show that state-of-the-art models are largely version-oblivious: performance degrades for evolving APIs, while for stable APIs it remains the same across versions. Moreover, simply specifying the target version provides no benefit, while relevant documentation significantly boosts models' accuracy. These findings highlight a systematic limitation of current training paradigms and motivate new approaches for temporally grounded knowledge in code generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
