More Than Catastrophic Forgetting: Integrating General Capabilities For Domain-Specific LLMs
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Tongyi Lab, Alibaba Group(阿里巴巴集团通义实验室);Polytechnic Institute, Zhejiang University(浙江大学理工学院)
The performance on general tasks decreases after Large Language Models (LLMs) are fine-tuned on domain-specific tasks, the phenomenon is known as Catastrophic Forgetting (CF). However, this paper presents a further challenge for real application of domain-specific LLMs beyond CF, called General Capabilities Integration (GCI), which necessitates the integration of both the general capabilities and domain knowledge within a single instance. The objective of GCI is not merely to retain previously acquired general capabilities alongside new domain knowledge, but to harmonize and utilize both sets of skills in a cohesive manner to enhance performance on domain-specific tasks. Taking legal domain as an example, we carefully design three groups of training and testing tasks without lacking practicability, and construct the corresponding datasets. To better incorporate general capabilities across domain-specific scenarios, we introduce ALoRA, which utilizes a multi-head attention module upon LoRA, facilitating direct information transfer from preceding tokens to the current one. This enhancement permits the representation to dynamically switch between domain-specific knowledge and general competencies according to the attention. Extensive experiments are conducted on the proposed tasks. The results exhibit the significance of our setting, and the effectiveness of our method.
An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
Pattern Recognition Center, WeChat AI, Tencent Inc(腾讯公司微信AI模式识别中心);School of Engineering, Westlake University(西湖大学工程学院);Institute of Advanced Technology, Westlake Institute for Advanced Study(西湖高等研究院先进技术研究所)
AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention
School of Software Technology, Zhejiang University(浙江大学软件学院);College of Intelligent Robotics and Advanced Manufacturing, Fudan University(复旦大学智能机器人与先进制造学院);School of Artificial Intelligence, Sun Yat-Sen University(中山大学人工智能学院);Graduate School of Information Science, Hokkaido University(北海道大学信息科学研究院)
When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting
MPI for Intelligent Systems, Tübingen(马克斯·普朗克智能系统研究所(蒂宾根));Jinesis Lab, University of Toronto & Vector Institute(Jinesis实验室,多伦多大学与向量研究所);EuroSafeAI;Hector Foundation(赫克托基金会);Stanford University(斯坦福大学);ELLIS Institute Tübingen(ELLIS研究所(蒂宾根))
Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives
Tsinghua University(清华大学);Microsoft Research(微软研究院);Georgia Institute of Technology(佐治亚理工学院);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心)
Mitigating Catastrophic Forgetting in Multi-domain Chinese Spelling Correction by Multi-stage Knowledge Transfer Framework
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);State Key Lab of Software Development Environment, Beihang University(北京航空航天大学软件开发环境国家重点实验室);School of Intelligent Systems Engineering, Sun-Yat Sen University(中山大学智能工程学院);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东省人工智能与数字经济实验室(深圳))