用于词级时间戳标注的相对时间间隔表示：基于掩码训练
Relative Time Intervals Representation for Word-level Timestamping with Masked Training
浏览论文内容
中文总结 AI 辅助
本研究针对SpeechLLMs时间对齐输出能力不足的问题，提出用相对时间戳替代绝对时间戳，结合混合微调策略与掩码训练目标，在提升时间戳预测准确率的同时保持了语音转录性能。
中文摘要 AI 辅助
尽管语音大语言模型（SpeechLLMs）在语音理解与生成方面表现出色，但其细粒度、时间对齐输出的能力仍未得到充分探索。本研究解决这一缺口，使SpeechLLMs能够联合建模语音内容与时间结构，有效将其从“内容理解机器”转变为“感知时间的内容理解机器”。具体而言，我们用相对时间戳替代传统绝对时间戳，以实现更紧凑的词汇表与更强的泛化能力。为将时间戳预测能力高效注入预训练大语言模型，我们引入混合微调策略：对时间戳增强的嵌入层与语言模型头进行全参数微调，同时对解码器层进行LoRA微调。此外，我们设计了掩码时间戳训练目标，防止模型过度依赖真实时间戳，从而增强其对现实世界带噪标注的鲁棒性。大量实验表明，我们的方法在时间戳预测准确率上取得显著提升，同时保持了出色的语音转录性能。
英文摘要
Although Speech Large Language Models (SpeechLLMs) excel at speech understanding and generation, their capacity for fine-grained, temporally aligned outputs remains underexplored. Our work addresses this gap by enabling SpeechLLMs to jointly model speech content and temporal structure, effectively transforming them from ``content understanding machines" into ``temporal-aware content understanding machines". Specifically, we replace traditional absolute timestamps with relative timestamps, achieving a more compact vocabulary and stronger generalization capabilities. To efficiently infuse timestamp prediction ability into pre-trained large language models, we introduce a hybrid fine-tuning strategy: full-parameter fine-tuning of the timestamp-augmented embedding layer and language model head, combined with LoRA fine-tuning of the decoder layers. Moreover, we design a masked timestamp training objective, preventing the model from over-relying on ground-truth timestamps, and thereby enhancing robustness against noisy real-world annotations. Extensive experiments demonstrate that our approach achieves significant improvements in timestamp prediction accuracy while maintaining strong speech transcription performance.
发表机构
- Soochow University（苏州大学）
- University of Queenland（昆士兰大学）
- AISpeech Ltd（思必驰科技有限公司）
机构由 AI 辅助整理，请以论文原文为准。
