MergeRepair：关于合并代码大语言模型中任务专用适配器以实现自动程序修复的探索性研究
MergeRepair: An Exploratory Study on Merging Task-Specific Adapters in Code LLMs for Automated Program Repair
AI总结：
本文提出 MergeRepair，探索用 weight-averaging、ties、dare-ties 及持续合并方法融合代码 LLM 的任务适配器，以提升自动程序修复能力。
摘要：
大语言模型（LLM）已在程序修复、文档编写、代码重构、调试和测试等若干软件开发相关任务中展现出强大能力。然而，训练这些模型需要海量数据和大量计算资源。适配器是专门设计的小型模块，用于对 LLM 进行参数高效微调，以适配特定任务、领域或应用，而无需对整个模型进行大规模重新训练。这些适配器利用大模型已有的能力，为根据特定需求定制 LLM 提供了一种更高效的方式。模型（及适配器）合并已经成为一种技术，可用于开发一个能够处理多种任务的模型，并且只需极少训练甚至无需训练。尽管模型和适配器合并在自然语言处理、计算机视觉等领域已表现出良好性能，但其在软件工程任务中的适用性仍未得到充分研究。本文通过我们提出的方法 MergeRepair，考察合并适配器在软件工程语境下的有效性，并特别关注自动程序修复（APR）任务。具体而言，我们使用 weight-averaging、ties 和 dare-ties 三种不同的合并方法合并多个任务专用适配器，并评估合并后的适配器在 APR 任务上的性能。我们提出一种持续合并方法，这是一种顺序合并任务专用适配器的新方法，其中被合并适配器的顺序和权重具有重要作用。我们还将该方法的性能与一种基线方法进行比较，该基线由在不同适配器参数上施加等权重合并构成，其中所有适配器具有同等重要性。
英文摘要：
Large Language Models (LLMs) have shown high capabilities in several software development-related tasks such as program repair, documentation, code refactoring, debugging, and testing. However, training these models requires massive amount of data and significant computational resources. Adapters are specialized, small modules designed for parameter efficient fine-tuning of LLMs for specific tasks, domains, or applications without requiring extensive retraining of the entire model. These adapters offer a more efficient way to customize LLMs for particular needs, leveraging the pre-existing capabilities of the large model. Model (and adapter) merging have emerged as a technique to develop one model capable of multiple tasks, with minimal or no training required. Although model and adapter merging has shown promising performance in domains such as natural language processing and computer vision, its applicability to software engineering tasks remains underexplored. In this paper, we investigate the effectiveness of merged adapters within the context of software engineering, with a particular focus on the Automated Program Repair (APR) task, through our approach, MergeRepair. In particular, we merge multiple task-specific adapters using three different merging methods, including weight-averaging, ties, and dare-ties, and evaluate the performance of the merged adapter on the APR task. We introduce a continual merging approach, a novel method in which we sequentially merge the task-specific adapters where the order and weight of the merged adapters play a significant role. We further compare the performance of our approach with a baseline method consisting of equal-weight merging applied on parameters of different adapters, where all adapters are of equal importance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
