arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.01802cs.SDcs.AI
收藏

MOSS-Audio 技术报告

MOSS-Audio Technical Report

OpenMOSS Team(开放MOSS团队)
Chen Yang, Chufan Yu, Hanfu Chen, Jie Zhu, Jingqi Chen, Ke Chen, Wenxuan Wang, Yang Wang, Yaozhou Jiang, Yi Jiang, Zhengyuan Lin, Ziqi Chen, Zhaoye Fei, Chengha… 展开作者

Chen Yang, Chufan Yu, Hanfu Chen, Jie Zhu, Jingqi Chen, Ke Chen, Wenxuan Wang, Yang Wang, Yaozhou Jiang, Yi Jiang, Zhengyuan Lin, Ziqi Chen, Zhaoye Fei, Chenghao Liu, Donghua Yu, Jun Zhan, Kang Yu, Kexin Huang, Liwei Fan, Mingshu Chen, Qinyuan Cheng, Ruixiao Li, Shimin Li, Songlin Wang, Xingjian Zhao, Yang Gao, Yitian Gong, Yiyang Zhang, Zhe Xu, Xipeng Qiu

AI总结:

提出统一音频-语言模型 MOSS-Audio,通过 DeepStack 跨层特征注入和时间标记实现语音、环境声和音乐的理解,在音频字幕、时间感知问答、时间戳转录和音频推理任务上取得强性能。

摘要:

MOSS-Audio 是一个统一的音频-语言模型,用于语音、环境声和音乐的理解,支持音频字幕、时间感知问答、时间戳转录和基于音频的推理。MOSS-Audio 将专用音频编码器与模态适配器和大语言模型耦合:编码器产生 12.5 Hz 的时间表示,适配器将其投影到解码器空间,解码器生成自回归文本输出。两个设计选择是系统的核心: extbf{DeepStack 跨层特征注入},将来自多个编码器深度的声学信息暴露给解码器;以及 extbf{时间标记},通过在音频标记流中插入时间戳标记来提供显式的时间线索。在数据层面,我们设计了一个事件保留的音频标注流程,将原始音频在连贯的事件边界处分割,对语音、音乐和通用音频应用分支特定标注,并将结果合并为统一的字幕用于预训练。中间的分支特定字幕进一步保留以支持面向任务的 SFT 数据的构建。该模型在大规模音频-语言数据上进行预训练,结合时间感知目标以支持时间定位,然后进行多阶段后训练以增强指令遵循和基于音频的推理。我们发布了 4B 和 8B 两种变体,包括 Instruct 和 Thinking 配置。MOSS-Audio 在通用音频理解、语音字幕、ASR 和时间戳 ASR 上取得了强性能,使其成为未来语音代理的有前途的理解基础。

英文摘要:

MOSS-Audio is a unified audio-language model for speech, environmental sound, and music understanding, supporting audio captioning, time-aware question answering, timestamped transcription, and audio-grounded reasoning. MOSS-Audio couples a dedicated audio encoder with a modality adapter and a large language model: the encoder produces 12.5 Hz temporal representations, the adapter projects them into the decoder space, and the decoder generates autoregressive text outputs. Two design choices are central to the system: DeepStack cross-layer feature injection, which exposes the decoder to acoustic information from multiple encoder depths, and time markers, which provide explicit temporal cues by inserting timestamp markers into the audio-token stream. At the data level, we design an event-preserving audio annotation pipeline that segments raw audio at coherent event boundaries, applies branch-specific annotation to speech, music, and general audio, and merges the results into unified captions for pretraining. The intermediate branch-specific captions are further retained to support the construction of task-oriented SFT data. The model is pretrained on large-scale audio-language data, with time-aware objectives incorporated to support temporal grounding, and then undergoes multi-stage post-training to enhance instruction following and audio-grounded reasoning. We release 4B and 8B variants in both Instruct and Thinking configurations. MOSS-Audio achieves strong performance across general audio understanding, speech captioning, ASR, and timestamped ASR, positioning it as a promising understanding foundation for future voice agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑