CodeUpdateArena：API更新上的知识编辑基准测试
CodeUpdateArena: Benchmarking Knowledge Editing on API Updates
AI总结：
本文提出CodeUpdateArena基准，评估代码领域API更新知识编辑，发现现有LLMs和编辑技术难以有效纳入API更新。
摘要：
大型语言模型（LLMs）正越来越多地被用于合成和推理源代码。然而，这些模型知识的静态特性并不能反映这样一个事实：它们调用的库和API函数在不断演进，功能会被添加或更改。尽管已有许多基准测试评估LLMs如何生成代码，但此前尚无研究探讨如何更新LLMs关于代码API函数的知识。为了填补这一空白，我们提出了CodeUpdateArena，一个用于代码领域知识编辑的基准测试。我们基准测试中的一个实例由一个合成的API函数更新与一个使用该更新功能的程序合成示例配对组成；我们的目标是更新LLM，使其能够在推理时不提供更新文档的情况下解决该程序合成示例。与针对文本中编码事实的知识编辑相比，此处的成功更具挑战性：代码LLM必须正确推理修改后函数的语义，而不仅仅是复现其语法。我们的数据集构建过程是首先提示GPT-4生成原子化且可执行的函数更新。然后，对于每个更新，我们生成程序合成示例，其代码解决方案倾向于使用该更新。我们的基准测试涵盖了对来自七个不同Python包的54个函数的各种类型更新，总共有670个程序合成示例。我们的实验表明，将更新文档前置添加到开源代码LLMs（即DeepSeek、CodeLlama）中并不能使它们在解决问题时纳入这些更改，而现有的知识编辑技术也有很大的改进空间。我们希望我们的基准测试能激发代码LLMs中知识更新的新方法。
英文摘要：
Large language models (LLMs) are increasingly being used to synthesize and reason about source code. However, the static nature of these models' knowledge does not reflect the fact that libraries and API functions they invoke are continuously evolving, with functionality being added or changing. While numerous benchmarks evaluate how LLMs can generate code, no prior work has studied how an LLMs' knowledge about code API functions can be updated. To fill this gap, we present CodeUpdateArena, a benchmark for knowledge editing in the code domain. An instance in our benchmark consists of a synthetic API function update paired with a program synthesis example that uses the updated functionality; our goal is to update an LLM to be able to solve this program synthesis example without providing documentation of the update at inference time. Compared to knowledge editing for facts encoded in text, success here is more challenging: a code LLM must correctly reason about the semantics of the modified function rather than just reproduce its syntax. Our dataset is constructed by first prompting GPT-4 to generate atomic and executable function updates. Then, for each update, we generate program synthesis examples whose code solutions are prone to use the update. Our benchmark covers updates of various types to 54 functions from seven diverse Python packages, with a total of 670 program synthesis examples. Our experiments show that prepending documentation of the update to open-source code LLMs (i.e., DeepSeek, CodeLlama) does not allow them to incorporate changes for problem solving, and existing knowledge editing techniques also have substantial room for improvement. We hope our benchmark will inspire new methods for knowledge updating in code LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
