arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.25145cs.CVcs.LG
收藏

学习排名图注链以实现视频-文本对齐

Learning to Rank Caption Chains for Video-Text Alignment

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Amazon Prime Video(亚马逊Prime Video)

Ansel Blume, Burak Uzkent, Shalini Chaudhuri, Garin Kessler

AI总结:

本文提出通过重复图注退化生成大规模挑战性图注链,改进视频-文本对齐的排名优化方法,优于二元DPO并在长形式内容生成中表现更优,需对视觉编码器进行微调。

摘要:

直接偏好优化(DPO)是一种有效训练语言模型生成更受青睐响应的技术。然而,这种二元

英文摘要:

Direct preference optimization (DPO) is an effective technique to train language models to generate preferred over dispreferred responses. However, this binary "winner-takes-all" approach is suboptimal for vision-language models whose response quality is highly dependent on visual content. In particular, a response may still be faithful to the visual inputs even if it is less preferable than an alternative. The standard Bradley-Terry DPO formulation lacks this nuance, upweighting winning responses without sufficient regard for whether the "losing" response still maintains high visual fidelity. In this work, we investigate ranking optimization as an alternative that more precisely situates responses' faithfulness to visual inputs. We focus on video-text alignment using detailed video captions, proposing a method to generate challenging, totally ordered caption chains at scale through repeated caption degradation. Our results show ranking optimization outperforms binary DPO for long-form content generation and assessment, and importantly, we find that these approaches require finetuning of the vision encoder to be effective, challenging the view of DPO as purely a language-reweighting process.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑