arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.09054cs.SDcs.MM
收藏

HAFM:用于音乐伴奏生成的分层自回归基础模型

HAFM: Hierarchical Autoregressive Foundation Model for Music Accompaniment Generation

Zhejiang Lab(浙江实验室);University of Science and Technology of China(中国科学技术大学);Zhejiang International Studies University(浙江国际 Studies 大学)

Jian Zhu, Jianwei Cui, Yunlong Xue, Shihao Chen, Yubang Zhang, Cheng Luo, Jun Sun

AI总结:

提出HAFM分层自回归基础模型,通过双速率分词和三阶段级联生成框架,在MUSDB18上FAD达1.71,主观偏好率51.5%,优于基线。

摘要:

音乐伴奏生成旨在自动产生与给定人声输入在节奏、和声和音色上连贯的器乐伴奏,在个性化音乐创作、编曲辅助和音乐教育中有广泛应用。现有方法主要在符号域或依赖单阶段音频生成框架,通常存在高层语义结构建模不足、声学细节重建有限和条件可控性弱的问题。为解决这些局限,本文提出HAFM,一种用于人声条件音乐伴奏生成的分层自回归基础模型。该模型采用双速率分词策略,其中$50$ Hz HuBERT语义标记捕获高层音乐结构,$75$ Hz EnCodec声学标记编码细粒度声学内容,实现语义和声学表示的显式解耦。在此基础上,设计了三阶段级联生成框架,逐步生成语义标记、粗声学标记和细声学标记,从全局结构到局部细节细化伴奏。在MUSDB18数据集上的客观评估表明,完整的三阶段模型实现了1.71的Fr{é}chet音频距离(FAD)分数,相比两阶段基线(FAD=2.10)相对改进18.6%。主观听测表明,生成的伴奏在直接比较中达到51.5%的偏好率,优于真实伴奏,并在节奏对齐、和声兼容性和整体音乐连贯性上显著优于随机基线。源代码和演示见https://this URL。

英文摘要:

Music accompaniment generation aims to automatically produce instrumental accompaniments that are rhythmically, harmonically, and timbrally coherent with a given vocal input, with broad applications in personalized music creation, arrangement assistance, and music education. Existing approaches, primarily operating in the symbolic domain or relying on single-stage audio generation frameworks, commonly suffer from insufficient high-level semantic structure modeling, limited acoustic detail reconstruction, and weak conditional controllability. To address these limitations, this paper proposes HAFM, a Hierarchical Autoregressive Foundation Model for vocal-conditioned music accompaniment generation. The model employs a dual-rate tokenization strategy in which $50$ Hz HuBERT semantic tokens capture high-level musical structure and $75$ Hz EnCodec acoustic tokens encode fine-grained acoustic content, enabling explicit disentanglement of semantic and acoustic representations. Building on this foundation, a three-stage cascaded generation framework is designed to progressively generate semantic tokens, coarse acoustic tokens, and fine acoustic tokens, refining the accompaniment from global structure to local detail. . Objective evaluation on the MUSDB18 dataset demonstrates that the full three-stage model achieves a Fr{é}chet Audio Distance (FAD) score of 1.71, representing an 18.6% relative improvement over the two-stage baseline (FAD = 2.10). Subjective listening tests show that the generated accompaniments achieve a 51.5% preference rate against ground-truth accompaniments in head-to-head comparisons, and substantially outperform the random baseline in terms of rhythmic alignment, harmonic compatibility, and overall musical coherence. The source code and demo are available at https://github.com/HackerHyper/HAFM.git.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑