arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.17705cs.AIcs.CL
收藏

上下文注意力调制:迈向大型语言模型中的高效多任务适配

Contextual Attention Modulation: Towards Efficient Multi-Task Adaptation in Large Language Models

SCSE, Beihang University(北京航空航天大学系统工程学院);ERC of ACAT, MOE(教育部人工智能教育联合体);City University of Hong Kong(香港城市大学);Huawei Technologies Ltd.(华为技术有限公司);Key Lab of DIM, MIIT(信息通信技术重点实验室);SEM, Beihang University(北京航空航天大学电子机械学院);Zhejiang University of Technology(浙江工业大学)

Dayan Pan, Zhaoyang Fu, Jingyuan Wang, Xiao Han, Yue Zhu, Xiangyu Zhao

AI总结:

针对 LLM 多任务适配中知识保留与专业化难以兼顾的问题,提出 CAM 及 HyCAM 框架,通过共享与专用模块和动态路由实现高效适配,平均性能提升 3.65%。

摘要:

大型语言模型(LLM)具备显著的泛化能力,但在多任务适配方面存在困难,尤其是在平衡知识保留与任务特定专业化方面。传统微调方法会遭遇灾难性遗忘并消耗大量资源,而现有的参数高效方法在复杂多任务场景中表现欠佳。为解决这一问题,我们提出上下文注意力调制(Contextual Attention Modulation,CAM),这是一种动态调制 LLM 中自注意力模块表征的新颖机制。CAM 在增强任务特定特征的同时保留通用知识,从而促进更有效、更高效的适配。为实现有效的多任务适配,CAM 被集成到我们的混合上下文注意力调制(HyCAM)框架中,该框架将一个共享的全参数 CAM 模块与多个专门的轻量级 CAM 模块相结合,并通过动态路由策略加以增强,以实现自适应知识融合。在包括问答、代码生成和逻辑推理在内的异构任务上进行的大量实验表明,我们的方法显著优于现有方法,实现了 3.65% 的平均性能提升。为便于复现,已实现的代码和数据可在 https://github.com/Applied-Machine-Learning-Lab/HyCAM 获取。

英文摘要:

Large Language Models (LLMs) possess remarkable generalization capabilities but struggle with multi-task adaptation, particularly in balancing knowledge retention with task-specific specialization. Conventional fine-tuning methods suffer from catastrophic forgetting and substantial resource consumption, while existing parameter-efficient methods perform suboptimally in complex multi-task scenarios. To address this, we propose Contextual Attention Modulation (CAM), a novel mechanism that dynamically modulates the representations of self-attention modules in LLMs. CAM enhances task-specific features while preserving general knowledge, thereby facilitating more effective and efficient adaptation. For effective multi-task adaptation, CAM is integrated into our Hybrid Contextual Attention Modulation (HyCAM) framework, which combines a shared, full-parameter CAM module with multiple specialized, lightweight CAM modules, enhanced by a dynamic routing strategy for adaptive knowledge fusion. Extensive experiments on heterogeneous tasks, including question answering, code generation, and logical reasoning, demonstrate that our approach significantly outperforms existing approaches, achieving an average performance improvement of 3.65%. The implemented code and data are available to ease reproducibility at https://github.com/Applied-Machine-Learning-Lab/HyCAM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑