We study the fine-grained text-to-audio (T2A) generation task. While recent models can synthesize high-quality audio from text descriptions, they often lack precise control over attributes such as loudness, pitch, and sound events. Unlike prior approaches that retrain models for specific control types, we propose to train ControlNet models on top of pre-trained T2A backbones to achieve controllable generation over loudness, pitch, and event roll. We introduce two designs, T2A-ControlNet and T2A-Adapter, and show that the T2A-Adapter model offers a more efficient structure with strong control ability. With only 38M additional parameters, T2A-Adapter achieves state-of-the-art performance on the AudioSet-Strong in both event-level and segment-level F1 scores. We further extend this framework to audio editing, proposing T2A-Editor for removing and inserting audio events at time locations specified by instructions. Models, code, dataset pipelines, and benchmarks will be released to support future research on controllable audio generation and editing.
RFM-Editing: Rectified Flow Matching for Text-guided Audio Editing
Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, United Kingdom(视觉、语音和信号处理中心(CVSSP),英国萨里大学);College of Information and Electrical Engineering, China Agricultural University, China(信息与电气工程学院,中国农业大学);Fisheries College, Ocean University of China, China(海洋大学渔业学院,中国)
Unified Audio Generation and Editing via Joint Condition Modeling and Progressive Training
Department of Electronic Engineering and Information Science, University of Science and Technology of China(电子工程与信息科学系,中国科学技术大学);Tianjin Key Laboratory of Cognitive Computing and Application, School of Artificial Intelligence, Tianjin University(认知计算与应用重点实验室,天津大学人工智能学院);Institute of Artificial Intelligence of China Telecom (TeleAI)(中国电信人工智能研究院(TeleAI))
Fondazione Bruno Kessler(布鲁诺·凯塞勒基金会);Mila-Québec AI Institute(米拉-魁北克人工智能研究所);Concordia University(康考迪亚大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Laval University(拉瓦尔大学)
Beyond Coherence: Benchmarking Professional Editing-Technique Execution in Multi-Shot Audio-Video Generation
Shanghai Jiao Tong University(上海交通大学);Alibaba Group(阿里巴巴集团);Fudan University(复旦大学);UT Austin(德克萨斯大学奥斯汀分校);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所)
Arrange, Inpaint, and Refine: Steerable Long-term Music Audio Generation and Editing via Content-based Controls
New York University Shanghai(纽约大学上海分校);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Queen Mary University of London(伦敦大学玛丽皇后学院)
Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomy
University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);University of Texas at Dallas(德克萨斯大学达拉斯分校);Virginia Tech(弗吉尼亚理工大学);Texas A&M University(德克萨斯A&M大学);University of Southern California(南加州大学);Vanderbilt University(范德堡大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Adobe Research(Adobe研究院);Arizona State University(亚利桑那州立大学);University of Oregon(俄勒冈大学);Stanford University(斯坦福大学);Dolby Laboratories(杜比实验室);Cisco(思科)
UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion
The Chinese University of Hong Kong(香港中文大学);The Hong Kong Polytechnic University(香港理工大学);City University of Hong Kong(香港城市大学);The Hong Kong University of Science and Technology(香港科学与技术大学);Tsinghua University(清华大学);Huawei Research Hong Kong(华为香港研究)
REDDIT: Forgetting-Resistant Correction of Timestamp Drift in ASR via Replay-Based Distribution Editing
National Taiwan University(台湾大学);Carnegie Mellon University(卡内基梅隆大学);NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)(国立清华大学人工智能研究中心(NTU AI-CoRE))
RFM-Editing 2: Text-Guided Audio Editing with Rectified Flow Matching and Coarse-to-Fine Diffusion Transformers
Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey(萨里大学视觉、语音与信号处理中心);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院);Fisheries College, Ocean University of China(中国海洋大学水产学院);College of Information and Electrical Engineering, China Agricultural University(中国农业大学信息与电气工程学院)
Object-AVEdit: An Object-level Audio-Visual Editing Model
Institute of Artificial Intelligence (TeleAI)(人工智能研究院);China Telecom(中国电信);Gaoling School of Artificial Intelligence(东城区人工智能学院);Renmin University of China(中国人民大学);Beijing University of Posts and Telecommunications(北京邮电大学);Tencent Data Platform(腾讯数据平台);Tsinghua University(清华大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Huawei Noah’s Ark Lab(华为诺亚实验室);The University of Hong Kong(香港大学)
DirectAudioEdit: Inversion-Free Text-Guided Audio Editing via Diffusion Prediction Contrast
School of Computer Science and Engineering, Northeastern University, Shenyang, China(东北大学计算机科学与工程学院);Kunming University of Science and Technology(昆明理工大学);NiuTrans Research, Shenyang, China(新译研究)
AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities
Nanjing University(南京大学);Kuaishou Technology(快手科技);National University of Singapore(新加坡国立大学);Beijing University of Posts and Telecommunications(北京邮电大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Towards Automatic Evaluation and High-Quality Pseudo-Parallel Dataset Construction for Audio Editing: A Human-in-the-Loop Method
College of Computer Science, Nankai University(南开大学计算机科学学院);College of Electronic Information and Optical Engineering, Nankai University(南开大学电子信息与光工程学院)
EditGen: Harnessing Cross-Attention Control for Instruction-Based Auto-Regressive Audio Editing
Department of Informatics and Telecommunications, UoA NCSR "Demokritos"(信息与电信系,雅典国家科研中心"Demokritos");School of Electrical and Computer Engineering, NTUA(电气与计算机工程学院,国家技术研究院)