VideoPASTA：对视频-大语言模型对齐至关重要的7K偏好对
VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignment
AI总结：
VideoPASTA通过仅7,020个偏好对和直接偏好优化，针对性地对齐视频语言模型的空间、时间与跨帧关系，无需人工标注，即在多个基准上显著提升性能，实现即插即用。
摘要：
视频语言模型（Video-LLMs）在理解视频内容方面表现出色，但在空间关系、时间顺序和跨帧连续性方面存在不足。为解决这些局限，我们提出了VideoPASTA（具有时空跨帧对抗样本的偏好对齐），一个通过针对性偏好优化来增强视频语言模型的框架。VideoPASTA训练模型区分准确的视频表示与精心构造的、故意违反空间、时间或跨帧关系的对抗样本。仅使用7,020个偏好对和直接偏好优化，VideoPASTA使模型能够学习捕捉细粒度空间细节和长程时间动态的鲁棒表示。实验表明，VideoPASTA是模型无关的，并能显著提升性能，例如，在应用于各种最先进的视频语言模型时，在LongVideoBench上取得高达+3.8个百分点的提升，在VideoMME上取得+4.1的提升，在MVBench上取得+4.0的提升。这些结果表明，针对性的对齐，而非大规模预训练或架构修改，能有效解决核心的视频语言挑战。值得注意的是，VideoPASTA在无需任何人工标注或字幕的情况下实现了这些改进，仅依赖32帧采样。这种高效性使我们的方法成为一种可扩展的即插即用解决方案，能与现有模型无缝集成，同时保留其原有能力。
英文摘要：
Video-language models (Video-LLMs) excel at understanding video content but struggle with spatial relationships, temporal ordering, and cross-frame continuity. To address these limitations, we introduce VideoPASTA (Preference Alignment with Spatio-Temporal-Cross Frame Adversaries), a framework that enhances Video-LLMs through targeted preference optimization. VideoPASTA trains models to distinguish accurate video representations from carefully crafted adversarial examples that deliberately violate spatial, temporal, or cross-frame relationships. With only 7,020 preference pairs and Direct Preference Optimization, VideoPASTA enables models to learn robust representations that capture fine-grained spatial details and long-range temporal dynamics. Experiments demonstrate that VideoPASTA is model agnostic and significantly improves performance, for example, achieving gains of up to +3.8 percentage points on LongVideoBench, +4.1 on VideoMME, and +4.0 on MVBench, when applied to various state-of-the-art Video-LLMs. These results demonstrate that targeted alignment, rather than massive pretraining or architectural modifications, effectively addresses core video-language challenges. Notably, VideoPASTA achieves these improvements without any human annotation or captioning, relying solely on 32-frame sampling. This efficiency makes our approach a scalable plug-and-play solution that seamlessly integrates with existing models while preserving their original capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
