arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.08285cs.CLcs.AI
收藏

基于层级剪枝的参数高效微调在自由文本序列到序列建模中的应用

Parameter-Efficient Fine-Tuning with Layer Pruning on Free-Text Sequence-to-Sequence Modeling

Hainan University(海南大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Guangzhou University(广州大学)

Yunqi Zhu, Xuebing Yang, Yuanyuan Wu, Wensheng Zhang

AI总结:

本文提出一种集成 LoRA 与结构化层级剪枝的参数高效微调框架,在大幅降低 GPU 内存和训练时间的同时,有效保持了自由文本序列到序列任务的生成质量。

摘要:

语言模型规模的不断增大,使得人们对参数高效微调(如 LoRA)产生了浓厚的研究兴趣,该方法冻结预训练模型,并为多个下游任务(如摘要生成、问答和翻译)注入小规模可训练参数。为进一步提升微调效率,我们提出了一种集成 LoRA 与结构化层级剪枝的框架。该集成框架在两个基于 MIMIC-IV-Note 创建的去标识化医疗报告摘要数据集和两个公开医疗对话数据集上进行了验证。通过微调原始模型 0.6% 的参数并剪枝超过 30% 的 Transformer 层,我们的框架能减少 50% 的 GPU 内存使用并使训练阶段提速 100%,同时在自由文本序列到序列任务中保留超过 92% 的生成质量。

英文摘要:

The increasing size of language models raises great research interests in parameter-efficient fine-tuning such as LoRA that freezes the pre-trained model, and injects small-scale trainable parameters for multiple downstream tasks (e.g., summarization, question answering and translation). To further enhance the efficiency of fine-tuning, we propose a framework that integrates LoRA and structured layer pruning. The integrated framework is validated on two created deidentified medical report summarization datasets based on MIMIC-IV-Note and two public medical dialogue datasets. By tuning 0.6% parameters of the original model and pruning over 30% Transformer-layers, our framework can reduce 50% of GPU memory usage and speed up 100% of the training phase, while preserving over 92% generation qualities on free-text sequence-to-sequence tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑