持续监督微调通过负监督匹配多模态RLHF
Continual SFT Matches Multimodal RLHF with Negative Supervision
AI总结：
本文提出负监督微调（nSFT）方法，通过解耦多模态RLHF中的负监督信息并用简单SFT损失持续对齐视觉语言模型，在更节省内存的同时取得与多模态RLHF相当的效果。
摘要：
多模态RLHF通常在监督微调（SFT）阶段之后进行，以持续提升视觉语言模型（VLMs）的理解能力。传统观点认为，在这一偏好对齐阶段，多模态RLHF优于持续SFT。本文观察到，多模态RLHF的内在价值在于其负监督，即被拒绝响应的logit。因此，我们提出了一种新颖的负监督微调（nSFT）方法，充分挖掘这些信息。我们的nSFT将RLHF范式中的负监督解耦出来，并用简单的SFT损失持续对齐VLMs。与严格要求2个（如DPO）或4个（如PPO）大型VLMs的多模态RLHF相比，该方法更节省内存。通过在不同数据集来源、基础VLMs和评估指标上与多种多模态RLHF方法进行比较，nSFT的有效性得到了严格证明。此外，我们还提供了丰富的消融实验来支持我们的假设。我们希望本文能激发进一步的研究，以恰当地对齐大型视觉语言模型。
英文摘要：
Multimodal RLHF usually happens after supervised finetuning (SFT) stage to continually improve vision-language models' (VLMs) comprehension. Conventional wisdom holds its superiority over continual SFT during this preference alignment stage. In this paper, we observe that the inherent value of multimodal RLHF lies in its negative supervision, the logit of the rejected responses. We thus propose a novel negative supervised finetuning (nSFT) approach that fully excavates these information resided. Our nSFT disentangles this negative supervision in RLHF paradigm, and continually aligns VLMs with a simple SFT loss. This is more memory efficient than multimodal RLHF where 2 (e.g., DPO) or 4 (e.g., PPO) large VLMs are strictly required. The effectiveness of nSFT is rigorously proved by comparing it with various multimodal RLHF approaches, across different dataset sources, base VLMs and evaluation metrics. Besides, fruitful of ablations are provided to support our hypothesis. We hope this paper will stimulate further research to properly align large vision language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
