TASTE-Streaming: 向可流式传输的文本对齐语音标记化与嵌入迈进
TASTE-Streaming: Towards Streamable Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling
AI总结：
本文提出TASTE-S，通过集成CTC ASR模块和重设计解码器，实现流式传输的文本对齐语音标记化与嵌入，减少延迟并保持鲁棒性。
摘要：
文本-语音联合 spoken language modeling (SLM) 目标是实现自然智能的语音交互，但开发此类系统可能面临模态不匹配问题：语音单元序列远长于文本标记。先前工作通过文本对齐标记化和嵌入（TASTE）缩小这一差距，生成与文本对应长度的语音标记。然而，依赖外部ASR系统和非因果解码器限制了流式传输应用。为解决此限制，我们提出TASTE-S，一种适用于实时使用的可流式传输TASTE扩展。TASTE-S将基于CTC的ASR模块集成到编码器中，实现即时双模态编码。我们还重新设计单元解码器以实现即用即解码。通过联合训练，我们证明TASTE-S在性能上与TASTE相当，同时显著降低延迟。进一步研究显示，TASTE-S在转录鲁棒性方面保持稳定，并支持长形式编码和解码。
英文摘要：
Text-speech joint spoken language modeling (SLM) aims at natural and intelligent speech-based interactions, but developing such a system may suffer from modality mismatch: speech unit sequences are much longer than text tokens. Prior work reduces this gap with text-aligned tokenization and embedding (TASTE), producing speech tokens that align in lengths with their textual counterparts. However, the dependence on an external ASR system and the use of a non-causal decoder limits streaming use. To address this limitation, we propose TASTE-S, a streamable extension of TASTE suitable for real-time usage. TASTE-S integrates a CTC-based ASR module into the encoder for instant dual-modality encoding. We also redesign the unit decoder to enable on-the-fly decoding. With joint training, we show that TASTE-S matches TASTE's performance while significantly reducing latency. Further investigations reveal that TASTE-S remains robust to transcriptions and enables long-form encoding and decoding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
