抗缺失的视频增强多模态不流利检测
Missingness-resilient Video-enhanced Multimodal Disfluency Detection
AI总结:
本文提出结合音频与视频的多模态语音不流利检测方法,通过权重共享、模态无关编码器和韧性融合应对推理时视频缺失,并在五项任务上显著优于仅音频方法。
摘要:
现有的大多数语音不流利检测技术仅依赖声学数据。在本研究中,我们提出了一种实用的多模态不流利检测方法,利用可用的视频数据与音频。我们整理了一个音视频数据集,并提出一种新颖的融合技术,该技术采用统一的权重共享、模态无关编码器来学习时间和语义上下文。我们的韧性设计能够适应现实场景,即在推理过程中视频模态有时可能缺失。当两种模态都确定完整时,我们还提出了替代融合策略。在五项不流利检测任务的实验中,我们的统一多模态方法显著优于仅音频的单模态方法:当视频和音频模态始终可用时,平均绝对提升为10%(即增加10个百分点);即使一半样本的视频模态缺失,仍取得7%的提升。
英文摘要:
Most existing speech disfluency detection techniques only rely upon acoustic data. In this work, we present a practical multimodal disfluency detection approach that leverages available video data together with audio. We curate an audiovisual dataset and propose a novel fusion technique with unified weight-sharing modality-agnostic encoders to learn the temporal and semantic context. Our resilient design accommodates real-world scenarios where the video modality may sometimes be missing during inference. We also present alternative fusion strategies when both modalities are assured to be complete. In experiments across five disfluency-detection tasks, our unified multimodal approach significantly outperforms Audio-only unimodal methods, yielding an average absolute improvement of 10% (i.e., 10 percentage point increase) when both video and audio modalities are always available, and 7% even when video modality is missing in half of the samples.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。