DeSTA：通过描述性语音-文本对齐增强语音语言模型
DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignment
AI总结：
本文提出DeSTA方法，利用语音字幕生成对齐语音与文本模态，使语音语言模型能理解语言及非语言特征，在Dynamic-SUPERB基准上泛化优异且无需微调即具零样本指令遵循能力。
摘要：
近期的语音语言模型（SLM）通常结合预训练语音模型来扩展大型语言模型（LLM）的能力。本文提出一种描述性语音-文本对齐方法，利用语音字幕生成弥合语音与文本模态间的差距，使SLM能解释并生成全面的自然语言描述，从而促进其理解语音中语言与非语言特征的能力。借助该方法增强后，我们的模型在Dynamic-SUPERB基准上表现优异，尤其在泛化至未见任务时。此外，我们发现该对齐模型无需显式语音指令微调即具备零样本指令遵循能力。这些发现凸显了通过引入丰富描述性语音字幕重塑指令遵循SLM的潜力。
英文摘要：
Recent speech language models (SLMs) typically incorporate pre-trained speech models to extend the capabilities from large language models (LLMs). In this paper, we propose a Descriptive Speech-Text Alignment approach that leverages speech captioning to bridge the gap between speech and text modalities, enabling SLMs to interpret and generate comprehensive natural language descriptions, thereby facilitating the capability to understand both linguistic and non-linguistic features in speech. Enhanced with the proposed approach, our model demonstrates superior performance on the Dynamic-SUPERB benchmark, particularly in generalizing to unseen tasks. Moreover, we discover that the aligned model exhibits a zero-shot instruction-following capability without explicit speech instruction tuning. These findings highlight the potential to reshape instruction-following SLMs by incorporating rich, descriptive speech captions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
