面向双语NVV感知自动语音识别的源自适应数据策展

中文摘要 AI 辅助

非语言发声（NVV），如笑声、叹息、呼吸和咳嗽，传达了情感和互动信息，而传统的自动语音识别（ASR）系统通常会丢弃这些信息。我们提出了一个面向ISCSLP 2026上NVVSpeech挑战赛赛道1的双语普通话音-英语系统，该赛道要求在词汇内容及其转录相对位置上联合转录16种NVV类别。我们的NVV感知Whisper通过检查点兼容的词汇表重映射来适配Whisper-medium，使得词汇标记和内联NVV标签能够在统一的自回归序列中被解码，而无需扩展词汇表。为了提供可靠且多样化的监督，我们进一步引入了一种源自适应数据策展策略，该策略通过声学增强和多模态LLM过滤来精炼公共NVV语料库，同时通过自动化预处理和标注从野外媒体中挖掘自发NVV。在官方双语评估协议下，所提出的系统将最终得分从33.32提高到53.61，消融实验证实了所提出的数据策展组件的互补优势。

英文摘要

Nonverbal vocalizations (NVVs), such as laughter, sighs, breaths, and coughs, convey affective and interactional information that conventional automatic speech recognition (ASR) systems often discard. We present a bilingual Mandarin-English system for Track 1 of the NVVSpeech Challenge at ISCSLP 2026, which requires joint transcription of lexical content and 16 NVV categories at their transcript-relative positions. Our NVV-Aware Whisper adapts Whisper-medium through checkpoint-compatible vocabulary remapping, enabling lexical tokens and inline NVV tags to be decoded within a unified autoregressive sequence without expanding the vocabulary. To provide reliable and diverse supervision, we further introduce a source-adaptive data curation strategy that refines public NVV corpora through acoustic augmentation and multimodal LLM filtering, while mining spontaneous NVVs from in-the-wild media through automated preprocessing and annotation. Under the official bilingual evaluation protocol, the proposed system improves final score from 33.32 to 53.61, with ablations confirming the complementary benefits of the proposed data-curation components.

发表机构

• Northwestern Polytechnical University（西北工业大学）
• Shenzhen Pimei Technology（深圳品梅科技）
• Yutuzhineng（语图智能）
• Nanjing University（南京大学）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

ISCSLP 2026