arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.17830cs.CL
收藏

不止灾难性遗忘:为领域专用大语言模型整合通用能力

More Than Catastrophic Forgetting: Integrating General Capabilities For Domain-Specific LLMs

College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Tongyi Lab, Alibaba Group(阿里巴巴集团通义实验室);Polytechnic Institute, Zhejiang University(浙江大学理工学院)

Chengyuan Liu, Yangyang Kang, Shihang Wang, Lizhi Qing, Fubang Zhao, Changlong Sun, Kun Kuang, Fei Wu

AI总结:

本文针对领域专用 LLM 微调中的通用能力整合问题,以法律领域构建任务与数据集,并提出基于 LoRA 多头注意力的 ALoRA 动态融合知识与能力。

摘要:

在大语言模型(LLM)针对领域专用任务进行微调后,其在通用任务上的性能会下降,这一现象被称为灾难性遗忘(CF)。然而,本文提出了领域专用 LLM 在真实应用中超越 CF 的进一步挑战,称为通用能力整合(GCI),它要求在单个实例中同时整合通用能力与领域知识。GCI 的目标不仅是在获得新领域知识的同时保留此前习得的通用能力,还要以协调一致的方式融合并运用这两类技能,以提升领域专用任务上的性能。以法律领域为例,我们精心设计了三组兼具实用性的训练与测试任务,并构建了相应数据集。为了更好地将通用能力融入领域专用场景,我们提出 ALoRA,该方法在 LoRA 之上使用多头注意力模块,促进信息从前置 token 直接传递到当前 token。这一增强使表示能够根据注意力在领域专用知识与通用能力之间动态切换。我们在所提出的任务上开展了大量实验,结果展示了我们设定的重要性以及方法的有效性。

英文摘要:

The performance on general tasks decreases after Large Language Models (LLMs) are fine-tuned on domain-specific tasks, the phenomenon is known as Catastrophic Forgetting (CF). However, this paper presents a further challenge for real application of domain-specific LLMs beyond CF, called General Capabilities Integration (GCI), which necessitates the integration of both the general capabilities and domain knowledge within a single instance. The objective of GCI is not merely to retain previously acquired general capabilities alongside new domain knowledge, but to harmonize and utilize both sets of skills in a cohesive manner to enhance performance on domain-specific tasks. Taking legal domain as an example, we carefully design three groups of training and testing tasks without lacking practicability, and construct the corresponding datasets. To better incorporate general capabilities across domain-specific scenarios, we introduce ALoRA, which utilizes a multi-head attention module upon LoRA, facilitating direct information transfer from preceding tokens to the current one. This enhancement permits the representation to dynamically switch between domain-specific knowledge and general competencies according to the attention. Extensive experiments are conducted on the proposed tasks. The results exhibit the significance of our setting, and the effectiveness of our method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑