arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.05682cs.CVcs.LG
收藏

VMDT:解码视频基础模型的可信度

VMDT: Decoding the Trustworthiness of Video Foundation Models

University of California, Berkeley(加州大学伯克利分校);University of California, Santa Cruz(加州大学圣克ruz分校);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Chicago(芝加哥大学);Amazon(亚马逊);Information Commissioner’s Office(信息专员办公室)
Yujin Potter, Zhun Wang, Nicholas Crispino, Kyle Montgomery, Alexander Xiong, Ethan Y. Chang, Francesco Pinto, Yuqi Chen, Rahul Gupta, Morteza Ziyadi, Christos … 展开作者

Yujin Potter, Zhun Wang, Nicholas Crispino, Kyle Montgomery, Alexander Xiong, Ethan Y. Chang, Francesco Pinto, Yuqi Chen, Rahul Gupta, Morteza Ziyadi, Christos Christodoulopoulos, Bo Li, Chenguang Wang, Dawn Song

AI总结:

本文提出首个统一评估平台 VMDT,系统评估视频基础模型在安全性、幻觉、公平性、隐私和对抗鲁棒性上的可信度,揭示现有模型的系统性缺陷并指出改进方向。

摘要:

随着基础模型日益复杂,确保其可信度变得愈发关键;然而,与文本和图像不同,视频模态仍缺乏全面的可信度基准。我们提出 VMDT(Video-Modal DecodingTrust),这是首个统一平台,用于在五个关键可信度维度上评估文本到视频(T2V)和视频到文本(V2T)模型:安全性、幻觉、公平性、隐私和对抗鲁棒性。通过使用 VMDT 对 7 个 T2V 模型和 19 个 V2T 模型进行广泛评估,我们发现了若干重要见解。例如,所有被评估的开源 T2V 模型都无法识别有害查询,并经常生成有害视频,同时与图像模态模型相比表现出更高程度的不公平性。在 V2T 模型中,不公平性和隐私风险随规模增大而上升,而幻觉和对抗鲁棒性则有所改善——尽管整体性能仍然较低。独特的是,安全性与模型规模没有相关性,这意味着当前安全水平由规模以外的因素决定。我们的发现凸显了开发更鲁棒、更可信的视频基础模型的迫切需要,而 VMDT 为衡量和跟踪实现这一目标的进展提供了系统性框架。代码可在 https://sunblaze-ucb.github.io/VMDT-page/ 获取。

英文摘要:

As foundation models become more sophisticated, ensuring their trustworthiness becomes increasingly critical; yet, unlike text and image, the video modality still lacks comprehensive trustworthiness benchmarks. We introduce VMDT (Video-Modal DecodingTrust), the first unified platform for evaluating text-to-video (T2V) and video-to-text (V2T) models across five key trustworthiness dimensions: safety, hallucination, fairness, privacy, and adversarial robustness. Through our extensive evaluation of 7 T2V models and 19 V2T models using VMDT, we uncover several significant insights. For instance, all open-source T2V models evaluated fail to recognize harmful queries and often generate harmful videos, while exhibiting higher levels of unfairness compared to image modality models. In V2T models, unfairness and privacy risks rise with scale, whereas hallucination and adversarial robustness improve -- though overall performance remains low. Uniquely, safety shows no correlation with model size, implying that factors other than scale govern current safety levels. Our findings highlight the urgent need for developing more robust and trustworthy video foundation models, and VMDT provides a systematic framework for measuring and tracking progress toward this goal. The code is available at https://sunblaze-ucb.github.io/VMDT-page/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑