ADEPT：通过自适应扩展与动态解耦微调进行持续预训练
ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuning
AI总结：
针对LLM持续预训练中的灾难性遗忘与知识纠缠问题，提出两阶段框架ADEPT，通过通用能力引导的选择性层扩展与自适应单元级解耦微调，在仅微调15%参数下显著提升领域适应效果并缓解遗忘。
摘要：
用于大型语言模型（LLM）领域适应的传统持续预训练（CPT）通常面临灾难性遗忘和领域容量受限的问题。现有策略采用层扩展，引入额外的可训练参数来容纳新知识。然而，均匀的扩展和更新仍然会将通用学习和领域学习纠缠在一起，削弱了其有效性。我们的初步研究表明，LLM表现出功能特化，其中层和单元差异化地编码通用关键能力，这表明参数扩展和优化应具备功能感知能力。随后我们提出了ADEPT，即用于持续预训练的自适应扩展与动态解耦微调，这是一个用于领域自适应CPT的两阶段框架。ADEPT首先执行通用能力引导的选择性层扩展，复制对通用领域最不关键的层以增加表示容量，同时最小化对通用知识的干扰。然后应用自适应单元级解耦微调，根据参数单元在通用领域的重要性，解耦扩展层内的参数单元，并分配不对称学习率以平衡知识注入与保留。在数学和医学基准上的实验表明，ADEPT在通用领域和目标领域上分别超越全参数CPT高达5.76%和5.58%，且仅微调15%的参数，训练时间不足50%。消融研究、理论分析和扩展研究进一步证明了针对性扩展和解耦优化的必要性，为高效且鲁棒的领域自适应CPT提供了新原则。代码开源于https://github.com/PuppyKnightUniversity/ADEPT
英文摘要：
Conventional continual pretraining (CPT) for large language model (LLM) domain adaptation often suffers from catastrophic forgetting and limited domain capacity. Existing strategies adopt layer expansion, introducing additional trainable parameters to accommodate new knowledge. However, the uniform expansion and updates still entangle general and domain learning, undermining its effectiveness. Our pilot studies reveal that LLMs exhibit functional specialization, where layers and units differentially encode general-critical capabilities, suggesting that parameter expansion and optimization should be function-aware. We then propose ADEPT, Adaptive Expansion and Dynamic Decoupled Tuning for continual pretraining, a two-stage framework for domain-adaptive CPT. ADEPT first performs General-Competence Guided Selective Layer Expansion, duplicating layers least critical for the general domain to increase representational capacity while minimizing interference with general knowledge. It then applies Adaptive Unit-Wise Decoupled Tuning, disentangling parameter units within expanded layers according to their general-domain importance and assigning asymmetric learning rates to balance knowledge injection and retention. Experiments on mathematical and medical benchmarks show that ADEPT outperforms full-parameter CPT by up to 5.76% on the general domain and 5.58% on the target domain with only 15% of parameters tuned and less than 50% training time. Ablation studies, theoretical analysis, and extended investigations further demonstrate the necessity of targeted expansion and decoupled optimization, providing new principles for efficient and robust domain-adaptive CPT. Our code is open-sourced at https://github.com/PuppyKnightUniversity/ADEPT
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
