This work introduces Text2FX, a method that leverages CLAP embeddings and differentiable digital signal processing to control audio effects, such as equalization and reverberation, using open-vocabulary natural language prompts (e.g., "make this sound in-your-face and bold"). Text2FX operates without retraining any models, relying instead on single-instance optimization within the existing embedding space, thus enabling a flexible, scalable approach to open-vocabulary sound transformations through interpretable and disentangled FX manipulation. We show that CLAP encodes valuable information for controlling audio effects and propose two optimization approaches using CLAP to map text to audio effect parameters. While we demonstrate with CLAP, this approach is applicable to any shared text-audio embedding space. Similarly, while we demonstrate with equalization and reverberation, any differentiable audio effect may be controlled. We conduct a listener study with diverse text prompts and source audio to evaluate the quality and alignment of these methods with human perception. Demos and code are available at anniejchu.github.io/text2fx.
FineLAP: Taming Heterogeneous Supervision for Fine-grained Language-Audio Pretraining
X-LANCE Lab, Shanghai Jiao Tong University(上海交通大学X-LANCE实验室);SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University(上海交通大学巴黎卓越工程师学院);Shanghai Innovation Institute(上海创新研究院);The Chinese University of Hong Kong(香港中文大学)
Nankai University(南开大学);TMCC, College of Computer Science(TMCC计算机学院);Hithink RoyalFlush AI Research Institute(Hithink RoyalFlush人工智能研究院);University of Exeter(埃克塞特大学)
A Reference-free Metric for Language-Queried Audio Source Separation using Contrastive Language-Audio Pretraining
Harbin Engineering University(哈尔滨工程大学);University of Technology Sydney(悉尼科技大学);University of Surrey(萨里大学);Harbin Institute of Technology(哈尔滨工业大学);University of Sheffield(谢菲尔德大学)
CLAP-Based Automatic Word Naming Recognition in Post-Stroke Aphasia
Signal Processing for Communication Group, Idiap Research Institute, Martigny, Switzerland(日内瓦瑞士IDIAP研究所信号处理与通信组);Faculty of Psychology and Educational Science, University of Geneva, Geneva, Switzerland(日内瓦大学心理学与教育科学学院)
DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioning
Shanghai Jiao Tong University(上海交通大学);SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University(上海交通大学巴黎卓越工程师学院);The Chinese University of Hong Kong(香港中文大学)
Exploring the User Experience of AI-Assisted Sound Searching Systems for Creative Workflows
Centre for Vision, Speech and Signal Processing, University of Surrey(视觉、语音和信号处理中心,萨里大学);Research and Development, British Broadcasting Corporation(英国广播公司研发部)
Guided by the Plan: Enhancing Faithful Autoregressive Text-to-Audio Generation with Guided Decoding
The Hong Kong Polytechnic University(香港理工大学);IROOTECH TECHNOLOGY(IROOTECH技术公司);Wolf 1069 b Lab, Sany Group(三一集团沃尔夫1069实验室);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Zhejiang University(浙江大学)
QAMRO: Quality-aware Adaptive Margin Ranking Optimization for Human-aligned Assessment of Audio Generation Systems
Dept. Computer Science and Information Engineering, National Taiwan Normal University, Taiwan(计算机科学与信息工程系,台湾国立台湾师范大学);Institute of Computer Science, Academia Sinica, Taiwan(学术院计算机科学研究所,台湾);United Link Co., Ltd., Taiwan(联合链接公司,台湾)
SmoothCLAP: Soft-Target Enhanced Contrastive Language\--Audio Pretraining for Affective Computing
CHI -- Chair of Health Informatics, TUM University Hospital, Munich, Germany(健康信息学系,塔尔博特大学医院,德国慕尼黑);Huawei, Netherlands(华为,荷兰);GLAM, Imperial College London, UK(GLAM,伦敦帝国学院,英国)
MATS: An Audio Language Model under Text-only Supervision
Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, China(中国科学院智能信息处理重点实验室(中国科学院));University of Chinese Academy of Sciences, China(中国科学院大学)