OpenDelta：一个用于预训练模型参数高效适配的即插即用库
OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models
AI总结：
本文提出开源库OpenDelta，通过即插即用方式实现多种delta调优方法，无需修改主干预训练模型代码即可高效适配大规模PTMs。
摘要：
大规模预训练模型（PTMs）的规模在适配下游任务时带来了显著挑战，因为全参数微调涉及高昂的优化开销和存储成本。为了解决这一问题，许多研究探索了参数高效调优方法，也称为“delta tuning”，这些方法只更新一小部分参数，即所谓的“delta模块”，同时保持主干模型的参数固定。然而，由于现有实现直接修改主干PTMs的代码，并为每个PTM硬编码特定的delta调优方法，delta tuning的实用性和灵活性受到了限制。在本文中，我们提出了OpenDelta，这是一个开源库，通过提供各种delta调优方法的即插即用实现来克服这些限制。我们的新技术消除了修改主干PTMs代码的需要，使OpenDelta能够兼容不同的、甚至是新型的PTMs。OpenDelta设计简洁、模块化且可扩展，为研究人员和从业者高效适配大型PTMs提供了一个全面的平台。
英文摘要：
The scale of large pre-trained models (PTMs) poses significant challenges in adapting to downstream tasks due to the high optimization overhead and storage costs associated with full-parameter fine-tuning. To address this, many studies explore parameter-efficient tuning methods, also framed as "delta tuning", which updates only a small subset of parameters, known as "delta modules", while keeping the backbone model's parameters fixed. However, the practicality and flexibility of delta tuning have been limited due to existing implementations that directly modify the code of the backbone PTMs and hard-code specific delta tuning methods for each PTM. In this paper, we present OpenDelta, an open-source library that overcomes these limitations by providing a plug-and-play implementation of various delta tuning methods. Our novel techniques eliminate the need to modify the backbone PTMs' code, making OpenDelta compatible with different, even novel PTMs. OpenDelta is designed to be simple, modular, and extensible, providing a comprehensive platform for researchers and practitioners to adapt large PTMs efficiently.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
