AudioCALM：面向通用音频生成的连续自回归语言建模
AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation
AI总结：
提出AudioCALM框架，通过连续音频潜变量的自回归预测和不对称模态专家架构，统一语音、声音和音乐生成，在三个任务上达到或超越专用模型性能。
摘要：
将语音、声音和音乐生成统一到一个模型中受到保真度、端到端训练、上下文条件化和可变长度合成之间权衡的阻碍，当前没有一种范式能完全解决这些问题。为应对这一挑战，我们提出AudioCALM，一个通用音频生成框架，将自回归（AR）下一个token预测从离散token扩展到连续音频潜变量：一个薄的流匹配头取代softmax来预测每个位置的整流流速度，并且一个块因果AR-Flow注意力模式产生任意长度的输出。多个音频生成任务的联合训练面临不对称的文本-音频不匹配：语音转录与特定时间跨度对齐，需要紧密的时间对齐注意力，而声音和音乐描述仅描述整体语义，依赖分散的整体注意力；混合两者会不成比例地降低声音和音乐生成。我们在两个层面解决这种不对称性：一种数据重构策略，将所有三个任务统一在单一描述风格的条件化接口下；以及一种新颖的架构——非对称模态专家混合（A-MoME），它为语音添加一个专用的残差专家，而声音和音乐共享主干，在非语音输入上不引入推理开销。实验结果表明，AudioCALM在语音、声音和音乐生成基准上匹配了特定模态的最先进性能，并优于先前的统一基线。
英文摘要：
Unifying speech, sound, and music generation in one model is hindered by tradeoffs between fidelity, end-to-end training, in-context conditioning, and variable-length synthesis that no current paradigm fully resolves. To address this challenge, we present AudioCALM, a universal audio generation framework that extends autoregressive (AR) next-token prediction from discrete tokens to continuous audio latents: a thin flow-matching head replaces the softmax to predict rectified-flow velocities at each position, and a block-causal AR-Flow attention pattern produces arbitrary-length output. Joint training of multiple audio generation tasks faces an asymmetric text--audio mismatch: speech transcripts align to specific time spans and demand tight, time-aligned attention, whereas sound and music captions describe only overall semantics and rely on diffuse, holistic attention; mixing the two disproportionately degrades sound and music generation. We address this asymmetry at two levels: a data reformulation strategy that unifies all three tasks under a single description-style conditioning interface, and a novel architecture Asymmetric Mixture-of-Modality-Experts (A-MoME), which adds a dedicated residual expert for speech while sound and music share the backbone, incurring no inference overhead on non-speech inputs. Experimental results demonstrate that AudioCALM matches modality-specific state-of-the-art and outperforms prior unified baselines on speech, sound, and music generation benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
