Listen Then See：基于说话者注意力的视频对齐
Listen Then See: Video Alignment with Speaker Attention
AI总结：
本文针对社交智能问答（SIQA）中文本模态支配导致视频模态被绕过的问题，提出一种跨模态对齐与表示融合方法，利用音频模态作为语言与视频的桥梁，在Social IQ 2.0数据集上达到82.06%准确率。
摘要：
基于视频的问答（Video QA）是一项具有挑战性的任务，而在解决社交智能问答（SIQA）时则变得更为复杂。SIQA需要上下文理解、时序推理和多模态信息的整合，但除此之外，它还需要处理细微的人类行为。此外，主要模态（文本）对其他模态的支配地位加剧了所涉及的复杂性。因此，有必要帮助该任务的次要模态与主要模态协同工作。在这项工作中，我们引入了一种跨模态对齐和随后的表示融合方法，在用于SIQA的Social IQ 2.0数据集上取得了最先进的结果（82.06%准确率）。我们的方法通过将音频模态作为与语言模态的桥梁，展现出增强的利用视频模态的能力。这通过减少现有技术中普遍存在的语言过拟合和由此产生的视频模态绕过问题，从而提升了性能。我们的代码和模型已在https://github.com/sts-vlcc/sts-vlcc上公开提供。
英文摘要：
Video-based Question Answering (Video QA) is a challenging task and becomes even more intricate when addressing Socially Intelligent Question Answering (SIQA). SIQA requires context understanding, temporal reasoning, and the integration of multimodal information, but in addition, it requires processing nuanced human behavior. Furthermore, the complexities involved are exacerbated by the dominance of the primary modality (text) over the others. Thus, there is a need to help the task's secondary modalities to work in tandem with the primary modality. In this work, we introduce a cross-modal alignment and subsequent representation fusion approach that achieves state-of-the-art results (82.06\% accuracy) on the Social IQ 2.0 dataset for SIQA. Our approach exhibits an improved ability to leverage the video modality by using the audio modality as a bridge with the language modality. This leads to enhanced performance by reducing the prevalent issue of language overfitting and resultant video modality bypassing encountered by current existing techniques. Our code and models are publicly available at https://github.com/sts-vlcc/sts-vlcc
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
