利用AI反馈强化学习调优视频大语言多模态模型
Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback
AI总结：
本文提出VLM-RLAIF方法，通过AI反馈强化学习实现视频与文本模态对齐，并引入上下文感知奖励建模，在多个视频基准上超越现有SFT模型。
摘要：
大语言模型的最新进展推动了视频大语言多模态模型（VLMMs）的发展。此前针对VLMMs的方法主要采用基于指令微调数据集的监督微调（SFT），将大语言模型与视觉编码器集成，并添加额外的可学习模块。视频与文本的多模态对齐仍然具有挑战性，主要原因是与纯文本数据相比，多模态指令微调数据的数量和质量不足。我们提出了一种新颖的对齐策略，利用多模态AI系统对自身进行监督，称为AI反馈强化学习（RLAIF），通过提供自偏好反馈来完善自身，并促进视频与文本模态的对齐。具体而言，我们提出上下文感知奖励建模，在生成偏好反馈时提供详细的视频描述作为上下文，以丰富对视频内容的理解。在多个视频基准上展示了性能提升，我们的多模态RLAIF方法VLM-RLAIF优于现有方法，包括SFT模型。我们承诺开源我们的代码、模型和数据集，以促进该领域的进一步研究。
英文摘要：
Recent advancements in large language models have influenced the development of video large multimodal models (VLMMs). The previous approaches for VLMMs involved Supervised Fine-Tuning (SFT) with instruction-tuned datasets, integrating LLM with visual encoders, and adding additional learnable modules. Video and text multimodal alignment remains challenging, primarily due to the deficient volume and quality of multimodal instruction-tune data compared to text-only data. We present a novel alignment strategy that employs multimodal AI system to oversee itself called Reinforcement Learning from AI Feedback (RLAIF), providing self-preference feedback to refine itself and facilitating the alignment of video and text modalities. In specific, we propose context-aware reward modeling by providing detailed video descriptions as context during the generation of preference feedback in order to enrich the understanding of video content. Demonstrating enhanced performance across diverse video benchmarks, our multimodal RLAIF approach, VLM-RLAIF, outperforms existing approaches, including the SFT model. We commit to open-sourcing our code, models, and datasets to foster further research in this area.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
