RLHF-V：通过细粒度校正性人类反馈的行为对齐实现可信多模态大语言模型
RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback
AI总结：
针对多模态大语言模型普遍存在的幻觉问题，该研究提出RLHF-V方法，通过细粒度校正性人类反馈的行为对齐及密集直接偏好优化，以少量数据大幅降低幻觉率，在开源模型可信性上达到SOTA，鲁棒性优于GPT-4V。
摘要：
多模态大语言模型（MLLMs）近期在多模态理解、推理和交互方面展现出出色能力，但现有模型普遍存在严重的幻觉问题，生成的文本无法基于关联图像实现事实性支撑，这一问题导致现有MLLMs不可信，难以在现实世界（尤其是高风险）场景中落地应用。为解决该挑战，我们提出RLHF-V，通过细粒度校正性人类反馈的行为对齐提升MLLM的可信性。具体而言，RLHF-V以幻觉的片段级校正形式收集人类偏好，并基于人类反馈执行密集直接偏好优化。在5个基准上开展的自动评估与人工评估综合实验表明，RLHF-V能显著提升MLLM行为的可信性，且在数据与计算效率上表现优异。值得注意的是，仅使用1.4k条标注数据样本，RLHF-V就将基础MLLM的幻觉率显著降低了34.8%，性能优于基于10k标注数据训练的同期模型LLaVA-RLHF。最终模型在开源MLLMs的可信性方面达到了当前最优性能，且在防止过度泛化引发的幻觉问题上，表现出比GPT-4V更强的鲁棒性。我们已在https://github.com/RLHF-V/RLHF-V开源代码、模型与数据。
英文摘要：
Multimodal Large Language Models (MLLMs) have recently demonstrated impressive capabilities in multimodal understanding, reasoning, and interaction. However, existing MLLMs prevalently suffer from serious hallucination problems, generating text that is not factually grounded in associated images. The problem makes existing MLLMs untrustworthy and thus impractical in real-world (especially high-stakes) applications. To address the challenge, we present RLHF-V, which enhances MLLM trustworthiness via behavior alignment from fine-grained correctional human feedback. Specifically, RLHF-V collects human preference in the form of segment-level corrections on hallucinations, and performs dense direct preference optimization over the human feedback. Comprehensive experiments on five benchmarks in both automatic and human evaluation show that, RLHF-V can enable substantially more trustworthy MLLM behaviors with promising data and computation efficiency. Remarkably, using 1.4k annotated data samples, RLHF-V significantly reduces the hallucination rate of the base MLLM by 34.8%, outperforming the concurrent LLaVA-RLHF trained on 10k annotated data. The final model achieves state-of-the-art performance in trustworthiness among open-source MLLMs, and shows better robustness than GPT-4V in preventing hallucinations aroused from over-generalization. We open-source our code, model, and data at https://github.com/RLHF-V/RLHF-V.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
