arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.00680cs.SDcs.AIeess.AS
收藏

编码器-解码器多模态说话人切换检测

Encoder-decoder multimodal speaker change detection

NAVER Corporation(NAVER公司);NAVER Cloud Corporation(NAVER云公司)

Jee-weon Jung, Soonshin Seo, Hee-Soo Heo, Geonmin Kim, You Jin Kim, Young-ki Kwon, Minjae Lee, Bong-Jin Lee

AI总结:

针对多模态说话人切换检测,本文提出融合音频与文本的编码器-解码器模型,采用1.5秒说话人嵌入和transformer解码器,取得最先进性能。

摘要:

说话人切换检测(SCD)任务旨在检测输入中说话人发生变化的位置,对多种应用都至关重要。已有若干研究仅使用音频输入来解决SCD任务,并表现出有限的性能。近年来,除音频外还利用文本模态的多模态SCD(MMSCD)模型展现出更好的性能。在本研究中,所提出模型建立在两个主要方案之上:一种用于模态融合的新颖机制,以及采用编码器-解码器架构。不同于以往从与单个单词对齐的极短音频片段中提取说话人嵌入的MMSCD工作,我们使用从1.5秒音频中提取的说话人嵌入。一个transformer解码器层进一步提升了仅编码器MMSCD模型的性能。所提出模型在报告SCD性能的研究中取得了最先进结果,并且也与近期通过人工转写将SCD与自动语音识别相结合的工作表现相当。

英文摘要:

The task of speaker change detection (SCD), which detects points where speakers change in an input, is essential for several applications. Several studies solved the SCD task using audio inputs only and have shown limited performance. Recently, multimodal SCD (MMSCD) models, which utilise text modality in addition to audio, have shown improved performance. In this study, the proposed model are built upon two main proposals, a novel mechanism for modality fusion and the adoption of a encoder-decoder architecture. Different to previous MMSCD works that extract speaker embeddings from extremely short audio segments, aligned to a single word, we use a speaker embedding extracted from 1.5s. A transformer decoder layer further improves the performance of an encoder-only MMSCD model. The proposed model achieves state-of-the-art results among studies that report SCD performance and is also on par with recent work that combines SCD with automatic speech recognition via human transcription.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑