建模多模态社会交互：密集对齐表示的新挑战与基线
Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations
AI总结：
本文针对多方交互建模的局限性，提出说话目标识别等三个新任务及数据标注，并构建利用密集对齐语言-视觉表示的多模态基线，有效建模细粒度社会交互。
摘要：
理解涉及语言和非语言线索的社会交互对于有效解读社会情境至关重要。然而，以往关于多模态社会线索的研究大多集中于单人的行为，或者依赖于未与多方环境中的话语对齐的整体视觉表示。因此，它们在建模多方交互的复杂动态方面存在局限性。本文引入了三个新的挑战性任务来建模多人之间的细粒度动态：说话目标识别、代词共指消解和提及玩家预测。我们提供了广泛的数据标注，以在社会推理游戏环境中构建这些新挑战。此外，我们提出了一种新颖的多模态基线，通过将视觉特征与其对应的话语同步，利用密集对齐的语言-视觉表示。这有助于同时捕捉与社会推理相关的语言和非语言线索。实验证明了所提出的具有密集对齐多模态表示的方法在建模细粒度社会交互方面的有效性。项目网站：https://sangmin-git.github.io/projects/MMSI。
英文摘要：
Understanding social interactions involving both verbal and non-verbal cues is essential for effectively interpreting social situations. However, most prior works on multimodal social cues focus predominantly on single-person behaviors or rely on holistic visual representations that are not aligned to utterances in multi-party environments. Consequently, they are limited in modeling the intricate dynamics of multi-party interactions. In this paper, we introduce three new challenging tasks to model the fine-grained dynamics between multiple people: speaking target identification, pronoun coreference resolution, and mentioned player prediction. We contribute extensive data annotations to curate these new challenges in social deduction game settings. Furthermore, we propose a novel multimodal baseline that leverages densely aligned language-visual representations by synchronizing visual features with their corresponding utterances. This facilitates concurrently capturing verbal and non-verbal cues pertinent to social reasoning. Experiments demonstrate the effectiveness of the proposed approach with densely aligned multimodal representations in modeling fine-grained social interactions. Project website: https://sangmin-git.github.io/projects/MMSI.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
