面向视频基础对话的组合式反事实对比学习（$C^3$）
$C^3$: Compositional Counterfactual Contrastive Learning for Video-grounded Dialogues
AI总结：
本文提出组合式反事实对比学习（$C^3$），通过事实/反事实采样和对比隐藏状态表示优化视频基础对话生成，并在AVSD基准上取得性能提升。
摘要：
视频基础对话系统旨在融合视频理解与对话理解，以生成同时关联对话和视频上下文的回复。现有方法大多采用深度学习模型，并在可用数据集相对较小的情况下取得了显著性能。然而，这些结果部分是通过利用数据集中的偏差而非发展多模态推理来实现的，导致泛化能力有限。本文提出一种新颖的组合式反事实对比学习（$C^3$）方法，在视频基础对话中构建事实样本与反事实样本之间的对比训练。具体而言，我们基于视频中的时间步和对话中的词元设计事实/反事实采样，并提出利用对象级或动作级方差的对比损失函数。与先前方法不同，我们关注组合式输出词元之间的对比隐藏状态表示，以在生成设置下优化表示空间。我们在音视频场景感知对话（AVSD）基准上取得了有前景的性能提升，并展示了该方法在视频与对话上下文基础定位方面的优势。
英文摘要：
Video-grounded dialogue systems aim to integrate video understanding and dialogue understanding to generate responses that are relevant to both the dialogue and video context. Most existing approaches employ deep learning models and have achieved remarkable performance, given the relatively small datasets available. However, the results are partly accomplished by exploiting biases in the datasets rather than developing multimodal reasoning, resulting in limited generalization. In this paper, we propose a novel approach of Compositional Counterfactual Contrastive Learning ($C^3$) to develop contrastive training between factual and counterfactual samples in video-grounded dialogues. Specifically, we design factual/counterfactual sampling based on the temporal steps in videos and tokens in dialogues and propose contrastive loss functions that exploit object-level or action-level variance. Different from prior approaches, we focus on contrastive hidden state representations among compositional output tokens to optimize the representation space in a generation setting. We achieved promising performance gains on the Audio-Visual Scene-Aware Dialogues (AVSD) benchmark and showed the benefits of our approach in grounding video and dialogue context.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
