Audio diffusion models can synthesize high-fidelity music from text, yet achieving fine-grained control over specific musical attributes remains challenging, as their internal mechanisms for representing high-level concepts are poorly understood. In this work, we use activation patching to demonstrate that recent audio diffusion architectures exhibit a semantic bottleneck, where a small, shared subset of consecutive attention layers controls distinct musical concepts, such as the presence of specific instruments, vocals, or genres. Building on this, we systematically evaluate a broad spectrum of steering paradigms, comparing activation steering against prompt-level, score-space, and weight-space interventions, analyzing the interaction between the steering mechanism and the intervention site. Our new benchmark, supported by an extensive user study, demonstrates that localized activation steering establishes a new state-of-the-art in audio concept modulation.
Steer-MoE: Efficient Audio-Language Alignment with a Mixture-of-Experts Steering Module
The University of Hong Kong, Fauclty of Science, Hong Kong(香港大学科学学院);Aix-Marseille University, Laboratoire Parole et Langage (LPL), France(艾克斯-马赛大学语言与言语实验室(LPL))
Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders
AI Foundation and Algorithm Lab(AI基础与算法实验室);National University of Science and Technology MISIS(科学与技术国立大学MISIS);National Research University Higher School of Economics(国家研究大学经济高等学院)
AR&D: A Framework for Retrieving and Describing Concepts for Interpreting AudioLLMs
Australian Institute for Machine Learning, University of Adelaide(澳大利亚机器学习研究所,阿德莱德大学);Dolby Laboratories(杜比实验室);School of Computer and Mathematical Sciences, University of Adelaide, Australia(计算机与数学科学学院,阿德莱德大学,澳大利亚)