CounterVid: 通过生成反事实视频缓解视频-语言模型中动作和时间幻觉
CounterVid: Counterfactual Video Generation for Mitigating Action and Temporal Hallucinations in Video-Language Models
AI总结：
CounterVid通过生成反事实视频缓解视频-语言模型中动作和时间幻觉问题，提出反事实视频生成框架和MixDPO方法，提升时间推理和动作识别性能。
摘要：
视频-语言模型（VLMs）在多模态理解方面表现出色，但在推理动作和时间顺序时仍容易产生幻觉。现有的缓解策略，如文本过滤或随机视频扰动，往往无法解决根本问题：过度依赖语言先验而非细粒度的视觉动态。我们提出了一种可扩展的反事实视频生成框架，该框架合成的视频仅在动作或时间结构上有所不同，同时保持场景上下文。我们的流程结合了多模态大语言模型用于动作提案和编辑指导，并结合基于扩散的图像和视频模型生成语义硬负样本。利用这一框架，我们构建了CounterVid，一个包含约26,000个偏好对的合成数据集，旨在动作识别和时间推理。我们进一步引入了MixDPO，一种统一的直接偏好优化方法，能够同时利用文本和视觉偏好。对Qwen2.5-VL进行MixDPO微调可带来一致的改进，特别是在时间顺序方面，并能有效转移到标准视频幻觉基准测试中。代码和模型将向公众开放。
英文摘要：
Video-language models (VLMs) achieve strong multimodal understanding but remain prone to hallucinations, especially when reasoning about actions and temporal order. Existing mitigation strategies, such as textual filtering or random video perturbations, often fail to address the root cause: over-reliance on language priors rather than fine-grained visual dynamics. We propose a scalable framework for counterfactual video generation that synthesizes videos differing only in actions or temporal structure while preserving scene context. Our pipeline combines multimodal LLMs for action proposal and editing guidance with diffusion-based image and video models to generate semantic hard negatives at scale. Using this framework, we build CounterVid, a synthetic dataset of ~26k preference pairs constructed from short counterfactual action clips and targeting both action recognition and controlled action-sequence ordering. We further introduce MixDPO, a unified Direct Preference Optimization approach that jointly leverages textual and visual preferences. Across Qwen2.5-VL and InternVL3 backbones, MixDPO substantially improves action recognition and temporal ordering, yields gains on most standard video hallucination benchmarks, and largely preserves general video understanding. Our source code, trained models, and dataset are available at https://aimagelab.github.io/CounterVid.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
