DMC$^3$：双模态反事实对比构建用于第一人称视频问答
DMC$^3$: Dual-Modal Counterfactual Contrastive Construction for Egocentric Video Question Answering
AI总结：
DMC$^3$通过双模态反事实对比构建方法提升第一人称视频问答性能，实现SOTA效果。
摘要：
第一人称视频问答（第一人称视频QA）在第一人称视频理解中起着重要作用，指的是基于第一视角视频回答问题。尽管现有方法通过预训练和微调范式取得了进展，但它们忽略了第一视角带来的独特挑战，例如理解多个事件和识别手-物体交互。为了解决这些挑战，我们提出了一种双模态反事实对比构建（DMC$^3$）框架，包含一个第一人称视频QA基线、一个反事实样本构建模块和一个涉及反事实样本的对比优化。具体而言，我们首先开发了一个反事实样本构建模块，通过事件描述改写和核心交互挖掘分别生成文本和视觉模态的正样本和负样本。然后，我们将这些样本与原始样本一起输入基线。最后，在涉及反事实样本的对比优化模块中，我们应用对比损失以最小化原始样本特征与正样本特征之间的距离，同时最大化与负样本的距离。实验表明，我们的方法在EgoTaskQA的normal和indirect分割上分别达到52.51%和46.04%，在QAEGO4D上达到13.2%，均达到最先进的性能。
英文摘要：
Egocentric Video Question Answering (Egocentric VideoQA) plays an important role in egocentric video understanding, which refers to answering questions based on first-person videos. Although existing methods have made progress through the paradigm of pre-training and fine-tuning, they ignore the unique challenges posed by the first-person perspective, such as understanding multiple events and recognizing hand-object interactions. To deal with these challenges, we propose a Dual-Modal Counterfactual Contrastive Construction (DMC$^3$) framework, which contains an egocentric videoqa baseline, a counterfactual sample construction module and a counterfactual sample-involved contrastive optimization. Specifically, We first develop a counterfactual sample construction module to generate positive and negative samples for textual and visual modalities through event description paraphrasing and core interaction mining, respectively. Then, We feed these samples together with the original samples into the baseline. Finally, in the counterfactual sample-involved contrastive optimization module, we apply contrastive loss to minimize the distance between the original sample features and the positive sample features, while maximizing the distance from the negative samples. Experiments show that our method achieve 52.51\% and 46.04\% on the \textit{normal} and \textit{indirect} splits of EgoTaskQA, and 13.2\% on QAEGO4D, both reaching the state-of-the-art performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
