CODESYNC：使大型语言模型与大规模动态代码演化保持同步
CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
AI总结：
本文提出CODESYNC数据引擎和CODESYNCBENCH基准，用于解决LLMs在动态代码演化中难以适应第三方库API更新的问题，实验表明现有模型和方法在此任务上表现不佳。
摘要：
大型语言模型（LLMs）在软件工程领域表现出卓越的性能，但在适应不断演化的代码知识方面面临挑战，尤其是在第三方库API频繁更新的情况下。这一局限性源于静态的预训练数据集，常常导致生成不可执行的代码或实现具有次优的安全性和效率。为此，本文介绍了CODESYNC，一个用于识别过时代码模式并从Python第三方库收集实时代码知识更新的数据引擎。基于CODESYNC，我们开发了CODESYNCBENCH，一个全面的基准测试，用于评估LLMs与代码演化保持同步的能力，该基准涵盖了来自六个Python库的220个API的真实世界更新。我们的基准提供了跨越三个评估任务的3,300个测试用例，以及一个包含2,200个训练样本的更新感知指令调优数据集。对14个最先进的LLMs进行的大量实验表明，即使有先进的知识更新方法（如DPO、ORPO和SimPO）的支持，它们也难以应对动态代码演化。我们相信，我们的基准可以为未来开发更有效的实时代码知识更新方法提供坚实的基础。实验代码和数据集可在https://github.com/Lucky-voyage/Code-Sync公开获取。
英文摘要：
Large Language Models (LLMs) have exhibited exceptional performance in software engineering yet face challenges in adapting to continually evolving code knowledge, particularly regarding the frequent updates of third-party library APIs. This limitation, stemming from static pre-training datasets, often results in non-executable code or implementations with suboptimal safety and efficiency. To this end, this paper introduces CODESYNC, a data engine for identifying outdated code patterns and collecting real-time code knowledge updates from Python third-party libraries. Building upon CODESYNC, we develop CODESYNCBENCH, a comprehensive benchmark for assessing LLMs' ability to stay synchronized with code evolution, which covers real-world updates for 220 APIs from six Python libraries. Our benchmark offers 3,300 test cases across three evaluation tasks and an update-aware instruction tuning dataset consisting of 2,200 training samples. Extensive experiments on 14 state-of-the-art LLMs reveal that they struggle with dynamic code evolution, even with the support of advanced knowledge updating methods (e.g., DPO, ORPO, and SimPO). We believe that our benchmark can offer a strong foundation for the development of more effective methods for real-time code knowledge updating in the future. The experimental code and dataset are publicly available at: https://github.com/Lucky-voyage/Code-Sync.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
