用于口语对话的基于神经音频编解码器和标签延迟训练的流式端点检测器
Streaming Endpointer for Spoken Dialogue using Neural Audio Codecs and Label-Delayed Training
AI总结：
本文提出利用流式低比特率神经音频编解码器特征和标签延迟训练方案，在160毫秒延迟下将端点检测截断错误降低42.7%（单流）和37.5%（双流），并集成到语音大模型中提升响应时间1200毫秒、减少截断错误35%。
摘要：
准确的、低延迟的端点检测对于有效的口语对话系统至关重要。虽然传统的端点检测器通常依赖于基于频谱的音频特征，但本工作提出了一种使用流式、低比特率神经音频编解码器（NAC）特征进行多轮对话的实时语音端点检测方法，该工作建立在神经音频编解码器的最新进展之上。为了进一步减少截断错误，我们引入了一种新颖的标签延迟训练方案。在固定的中位延迟为160毫秒的情况下，我们结合NAC和标签延迟的方法实现了显著的相对截断错误减少：与基线方法相比，单流端点检测器减少了42.7%，双流配置减少了37.5%。最后，我们展示了与基于编解码器的预训练语音大语言模型的高效集成，将其响应时间中位数提高了1200毫秒，并将其截断错误减少了35%。
英文摘要：
Accurate, low-latency endpointing is crucial for effective spoken dialogue systems. While traditional endpointers often rely on spectrum-based audio features, this work proposes real-time speech endpointing for multi-turn dialogues using streaming, low-bitrate Neural Audio Codec (NAC) features, building upon recent advancements in neural audio codecs. To further reduce cutoff errors, we introduce a novel label delay training scheme. At a fixed median latency of 160 ms, our combined NAC and label delay approach achieves significant relative cutoff error reductions: 42.7% for a single-stream endpointer and 37.5% for a two-stream configuration, compared to baseline methods. Finally, we demonstrate efficient integration with a codec-based pretrained speech large language model, improving its median response time by 1200 ms and reducing its cutoff error by 35%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
