保留预训练表征空间：论前缀调优在大型多模态模型中的有效性
Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Models
AI总结：
该研究针对大型多模态模型参数高效微调的作用机制问题，发现前缀调优可保留预训练表征空间，提出结合前缀调优与常规PEFT的PT-PEFT策略，在多任务上实现性能提升且保留表征空间。
摘要：
近年来，大型多模态模型（LMMs）正在革新机器与世界交互的方式，为各类多模态应用解锁了新的可能。为使LMMs适配下游任务，仅训练额外前缀令牌或模块的参数高效微调（PEFT）方法已得到广泛应用。然而，目前鲜有针对PEFT在LMMs中作用机制的分析。本文跳出这类方法通常关联的效率视角，深入探究了每种调优策略的优势与不足。我们首先发现，LoRA、Adapters等模型参数调优方法会扭曲预训练阶段习得的特征表征空间，限制对预训练知识的充分利用。我们还证实，前缀调优尽管在下游任务上表现较低，却能出色地保留表征空间。基于这些发现，我们提出了一种名为Prefix-Tuned PEFT（PT-PEFT，前缀调优型参数高效微调）的简单两步PEFT策略，该策略依次执行前缀调优与PEFT（即Adapter、LoRA），兼具二者优势。实验结果表明，与原生PEFT方法相比，PT-PEFT不仅提升了图像描述和视觉问答任务的性能，还有助于保留四个预训练模型的表征空间。
英文摘要：
Recently, we have observed that Large Multi-modal Models (LMMs) are revolutionizing the way machines interact with the world, unlocking new possibilities across various multi-modal applications. To adapt LMMs for downstream tasks, parameter-efficient fine-tuning (PEFT) which only trains additional prefix tokens or modules, has gained popularity. Nevertheless, there has been little analysis of how PEFT works in LMMs. In this paper, we delve into the strengths and weaknesses of each tuning strategy, shifting the focus from the efficiency typically associated with these approaches. We first discover that model parameter tuning methods such as LoRA and Adapters distort the feature representation space learned during pre-training and limit the full utilization of pre-trained knowledge. We also demonstrate that prefix-tuning excels at preserving the representation space, despite its lower performance on downstream tasks. These findings suggest a simple two-step PEFT strategy called Prefix-Tuned PEFT (PT-PEFT), which successively performs prefix-tuning and then PEFT (i.e., Adapter, LoRA), combines the benefits of both. Experimental results show that PT-PEFT not only improves performance in image captioning and visual question answering compared to vanilla PEFT methods but also helps preserve the representation space of the four pre-trained models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
