MCM-DPO：面向替代文本生成的多面跨模态直接偏好优化
MCM-DPO: Multifaceted Cross-Modal Direct Preference Optimization for Alt-text Generation
AI总结：
针对替代文本生成中标注噪声大、MLLMs上下文敏感度不足的问题，提出多面跨模态直接偏好优化方法MCM-DPO，构建TAlt、PAlt数据集，性能超越DPO和SFT，达到新SOTA。
摘要：
替代文本生成任务旨在生成简洁、与上下文相关的图像描述，帮助盲人和低视力用户获取在线图像内容。尽管大视觉语言模型（MLLMs）具备较强能力，但受用户标注噪声、标准不统一以及MLLMs对上下文信息不敏感等问题影响，替代文本生成的性能仍然有限。此前使用监督微调（SFT）微调MLLMs的研究效果不佳，因为SFT依赖准确的目标标注，而用户生成的替代文本往往存在缺陷。\n为解决这一问题，我们提出多面跨模态直接偏好优化（MCM-DPO），该方法无需精确标注，通过学习在偏好对中识别更优选项来提升替代文本生成质量。MCM-DPO从单偏好、配对偏好和多偏好三个维度优化偏好，覆盖文本、视觉和跨模态因素。鉴于替代文本领域缺乏高质量标注和偏好标注数据集，我们构建了两个大规模高质量数据集TAlt和PAlt，分别来自Twitter和Pinterest。这两个数据集包含20.2万条标注替代文本样本和1.8万条覆盖多种偏好维度的偏好对，旨在支持该领域的后续研究。实验结果表明，我们提出的MCM-DPO方法性能始终优于DPO和SFT，在替代文本生成任务上达到了新的最优水平。我们已在以下地址发布代码和数据：https://github.com/LVUGAI/MCM-DPO
英文摘要：
The alt-text generation task produces concise, context-relevant descriptions of images, enabling blind and low-vision users to access online images. Despite the capabilities of large vision-language models, alt-text generation performance remains limited due to noisy user annotations, inconsistent standards, and MLLMs' insensitivity to contextual information. Previous efforts to fine-tune MLLMs using supervised fine-tuning (SFT) have struggled, as SFT relies on accurate target annotations, which are often flawed in user-generated alt-text. To address this, we propose Multi-faceted Cross-modal Direct Preference Optimization (MCM-DPO), which improves alt-text generation by learning to identify better options in preference pairs without requiring precise annotations. MCM-DPO optimizes preferences across single, paired, and multi-preference dimensions, covering textual, visual, and cross-modal factors. In light of the scarcity of high-quality annotated and preference-labeled datasets for alt-text, we constructed two large-scale, high-quality datasets named TAlt and PAlt, sourced from Twitter and Pinterest. These datasets include 202k annotated alt-text samples and 18k preference pairs that cover diverse preference dimensions, aiming to support further research in this domain. Experimental results show that our proposed MCM-DPO method consistently outperforms both DPO and SFT, establishing a new state of the art in alt-text generation. We release the code and data here: https://github.com/LVUGAI/MCM-DPO
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
