arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.13263cs.CLcs.SDeess.AS
收藏

多模态一致性引导的参考自由数据选择用于ASR口音适应

Multimodal Consistency-Guided Reference-Free Data Selection for ASR Accent Adaptation

School of Computer Science and Technology（计算机科学与技术学院）；Xinjiang University（新疆大学）

Ligong Lei, Wenwen Lu, Xudong Pang, Zaokere Kadeer, Aishan Wumaier

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出一种多模态一致性引导的参考自由数据选择方法，用于提升ASR在口音适应中的性能，通过减少噪声伪标签和优化查询相关性，实现更高效的口音适应。

摘要：

自动语音识别（ASR）系统在口音语音上经常退化，因为声学-语音和语调变化导致与训练数据不匹配，使标记的口音适应成本高昂。然而，常见的伪标签选择启发式方法大多以文本为中心（例如，困惑度（PPL）过滤），并且倾向于选择流畅但声学不匹配的假设，导致在微调时产生误差放大。为了解决这个问题，我们引入了一种多模态一致性引导的、无参考的数据选择流程，用于在转导、无标签的协议下进行ASR口音适应。该流程首先基于子模ularity互信息进行目标感知的预选择步骤，以提高查询相关性和减少下游计算。然后通过扰动解码生成每个语音的多个伪转录文本，并使用两种无参考信号对每个假设进行评分：共享嵌入空间中的语音-文本对齐和预测的词错误率（WER）。一个简单的百分位数基于选择规则保留可靠的伪标签用于微调，同时丢弃噪声语音。在域内设置中，从3万个语音池中选择约1500个语音，达到10.91%的WER，接近使用3万个监督标签获得的10.45%的WER。在跨域设置中，与不匹配的候选池相比，一致性过滤的子集在强口音变化下避免了因未过滤的伪标签导致的退化，并在更强的ASR骨干网络上进行匹配小时实验，进一步证实了相对于随机采样和最近的选样基线的优势。

英文摘要：

Automatic speech recognition (ASR) systems often degrade on accented speech because acoustic-phonetic and prosodic shifts induce a mismatch to training data, making labeled accent adaptation costly. However, common pseudo-label selection heuristics are largely text-centric (e.g., perplexity (PPL) filtering) and can prefer fluent yet acoustically mismatched hypotheses, leading to error amplification when fine-tuning. To address this, we introduce a multimodal consistency-guided, reference-free data selection pipeline for ASR accent adaptation under a transductive, label-free protocol. The pipeline starts with a target-aware preselection step based on submodular mutual information to improve query relevance and reduce downstream computation. It then generates multiple pseudo-transcriptions per utterance via perturbation-based decoding and scores each hypothesis using two reference-free signals: speech--text alignment in a shared embedding space and predicted word error rate (WER). A simple percentile-based selection rule retains reliable pseudo-labels for fine-tuning while discarding noisy utterances. In an in-domain setting, selecting ~1.5k utterances from a 30k pool achieves 10.91% WER, close to 10.45% obtained using 30k supervised labels. In a cross-domain setting with a mismatched candidate pool, consistency-filtered subsets avoid the degradation caused by unfiltered pseudo-labels under strong accent shift, and matched-hour experiments on a stronger ASR backbone further confirm gains over random sampling and recent selection baselines.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑