V$^2$Dial：通过多模态专家统一视频与视觉对话
V$^2$Dial: Unification of Video and Visual Dialog via Multimodal Experts
AI总结：
本文提出 V$^2$Dial，利用专门多模态专家及匹配、对比学习统一图像与视频对话，并在 AVSD、VisDial 的四个基准上取得零样本和微调 SOTA。
摘要：
我们提出了 V$^2$Dial，这是一种新颖的基于专家的模型，专门面向同时处理图像和视频输入数据的多模态对话任务。当前的多模态模型主要关注较简单的任务（例如 VQA、VideoQA、视频-文本检索），往往忽视更具挑战性的对话类对应任务，如视频对话和视觉/图像对话。此外，尽管这两类对话任务具有明显的相似性，相关研究却彼此独立发展，从而限制了它们的应用潜力。为此，我们提出使用单一模型统一这两项任务：该模型首次通过将图像和视频路由至专门的专家来联合学习其空间与时间特征，并利用匹配学习和对比学习技术对这些特征进行对齐。此外，我们系统研究了两项任务之间的领域迁移，考察这些看似相关的任务能否以及在多大程度上从各自的训练数据中相互受益。在广泛使用的视频与视觉对话数据集 AVSD 和 VisDial 上进行的大量评估表明，我们的模型在零样本和微调设置下的四个基准上均取得了新的最先进结果。
英文摘要：
We present V$^2$Dial - a novel expert-based model specifically geared towards simultaneously handling image and video input data for multimodal conversational tasks. Current multimodal models primarily focus on simpler tasks (e.g., VQA, VideoQA, video-text retrieval) and often neglect the more challenging conversational counterparts, such as video and visual/image dialog. Moreover, works on both conversational tasks evolved separately from each other despite their apparent similarities limiting their applicability potential. To this end, we propose to unify both tasks using a single model that for the first time jointly learns the spatial and temporal features of images and videos by routing them through dedicated experts and aligns them using matching and contrastive learning techniques. Furthermore, we systemically study the domain shift between the two tasks by investigating whether and to what extent these seemingly related tasks can mutually benefit from their respective training data. Extensive evaluations on the widely used video and visual dialog datasets of AVSD and VisDial show that our model achieves new state-of-the-art results across four benchmarks both in zero-shot and fine-tuning settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
