睁开模型之眼：视频与上下文感知的多模态反馈预测
Open Your Model's Eyes: Video and Context-Aware Multimodal Backchannel Prediction
浏览论文内容
中文总结 AI 辅助
研究视频与上下文感知的多模态反馈预测问题，提出CAMA - BC框架，利用多层多模态对齐，分上下文对齐和反馈对齐两阶段，显著优于现有方法及简单多模态基线，尤其在识别复杂反馈上效果突出。
中文摘要 AI 辅助
反馈在自然人际互动中至关重要，它能传递诸如同理心和理解等倾听者状态。然而，当前方法仅依赖音频和文本，忽略了面部表情、手势等关键视觉线索以及更广泛的对话上下文，而这些对于准确预测是必要的。本文介绍了用于反馈预测的上下文感知多模态对齐（CAMA - BC），这是一个通过多层多模态对齐（MMA）利用视觉信息的新框架。对齐过程包括两个阶段，首先上下文对齐（MMA - CA）利用带视频的无标签对话捕获对话上下文，然后反馈对齐（MMA - BA）针对反馈预测微调表示。实验结果表明，CAMA - BC显著优于现有方法和简单多模态基线，在识别诸如同理心等复杂反馈方面特别有效。
英文摘要
Backchannels, which signal listener states like empathy and understanding, are fundamental to natural human interaction. However, current approaches rely solely on audio and text. This omits crucial visual cues, such as facial expressions and gestures, as well as broader conversational contexts, which are necessary for accurate prediction. In this paper, we introduce Context-Aware Multimodal Alignment for Backchannel Prediction (CAMA-BC), a novel framework that leverages visual information through Multi-Layer Multimodal Alignment (MMA). Our alignment process comprises two stages. First, Context Alignment (MMA-CA) utilizes unlabeled dialogues with videos to capture conversational contexts. Next, Backchannel Alignment (MMA-BA) fine-tunes the representations specifically for backchannel prediction. Experimental results show that CAMA-BC significantly outperforms both existing methods and simple multimodal baselines, with particular effectiveness in recognizing complex backchannels such as empathy.
发表机构
- Korea University（韩国大学）
- Kyung Hee University（庆熙大学）
机构由 AI 辅助整理，请以论文原文为准。
