We introduce WavCraft, a collective system that leverages large language models (LLMs) to connect diverse task-specific models for audio content creation and editing. Specifically, WavCraft describes the content of raw audio materials in natural language and prompts the LLM conditioned on audio descriptions and user requests. WavCraft leverages the in-context learning ability of the LLM to decomposes users' instructions into several tasks and tackle each task collaboratively with the particular module. Through task decomposition along with a set of task-specific models, WavCraft follows the input instruction to create or edit audio content with more details and rationales, facilitating user control. In addition, WavCraft is able to cooperate with users via dialogue interaction and even produce the audio content without explicit user commands. Experiments demonstrate that WavCraft yields a better performance than existing methods, especially when adjusting the local regions of audio clips. Moreover, WavCraft can follow complex instructions to edit and create audio content on the top of input recordings, facilitating audio producers in a broader range of applications. Our implementation and demos are available at this https://github.com/JinhuaLiang/WavCraft.
Unified Audio Generation and Editing via Joint Condition Modeling and Progressive Training
Department of Electronic Engineering and Information Science, University of Science and Technology of China(电子工程与信息科学系,中国科学技术大学);Tianjin Key Laboratory of Cognitive Computing and Application, School of Artificial Intelligence, Tianjin University(认知计算与应用重点实验室,天津大学人工智能学院);Institute of Artificial Intelligence of China Telecom (TeleAI)(中国电信人工智能研究院(TeleAI))
SALM: Spatial Audio Language Model with Structured Embeddings for Understanding and Editing
Institute of Acoustics, Chinese Academy of Sciences, China(中国科学院声学研究所);MiLM Plus, Xiaomi Inc., China(小米公司);Xi’an Jiaotong Liverpool University, China(西安交通大学利物浦大学);University of Chinese Academy of Sciences, China(中国科学院大学)
Object-AVEdit: An Object-level Audio-Visual Editing Model
Institute of Artificial Intelligence (TeleAI)(人工智能研究院);China Telecom(中国电信);Gaoling School of Artificial Intelligence(东城区人工智能学院);Renmin University of China(中国人民大学);Beijing University of Posts and Telecommunications(北京邮电大学);Tencent Data Platform(腾讯数据平台);Tsinghua University(清华大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Huawei Noah’s Ark Lab(华为诺亚实验室);The University of Hong Kong(香港大学)
Fondazione Bruno Kessler(布鲁诺·凯塞勒基金会);Mila-Québec AI Institute(米拉-魁北克人工智能研究所);Concordia University(康考迪亚大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Laval University(拉瓦尔大学)
EditGen: Harnessing Cross-Attention Control for Instruction-Based Auto-Regressive Audio Editing
Department of Informatics and Telecommunications, UoA NCSR "Demokritos"(信息与电信系,雅典国家科研中心"Demokritos");School of Electrical and Computer Engineering, NTUA(电气与计算机工程学院,国家技术研究院)
MeloDISinger: Melody-Aware & Duration-Preserving Singing Voice Editing with Audio Infilling
Graduate School of Artificial Intelligence, KAIST, South Korea(韩国釜山科学技术院人工智能研究生院);Graduate School of Culture Technology, KAIST, South Korea(韩国釜山科学技术院文化科技研究生院)
Speech Driven Video Editing via an Audio-Conditioned Diffusion Model
University of Galway(高威大学);University of Wrocław(弗罗茨瓦夫大学);Wrocław University of Science and Technology(弗罗茨瓦夫理工大学);Tooploox(Tooploox公司);Trinity College Dublin(都柏林圣三一学院)
Arrange, Inpaint, and Refine: Steerable Long-term Music Audio Generation and Editing via Content-based Controls
New York University Shanghai(纽约大学上海分校);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Queen Mary University of London(伦敦大学玛丽皇后学院)
Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomy
University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);University of Texas at Dallas(德克萨斯大学达拉斯分校);Virginia Tech(弗吉尼亚理工大学);Texas A&M University(德克萨斯A&M大学);University of Southern California(南加州大学);Vanderbilt University(范德堡大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Adobe Research(Adobe研究院);Arizona State University(亚利桑那州立大学);University of Oregon(俄勒冈大学);Stanford University(斯坦福大学);Dolby Laboratories(杜比实验室);Cisco(思科)
UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion
The Chinese University of Hong Kong(香港中文大学);The Hong Kong Polytechnic University(香港理工大学);City University of Hong Kong(香港城市大学);The Hong Kong University of Science and Technology(香港科学与技术大学);Tsinghua University(清华大学);Huawei Research Hong Kong(华为香港研究)
FluentEditor: Text-based Speech Editing by Considering Acoustic and Prosody Consistency
Inner Mongolia University(内蒙古大学);Zhejiang University(浙江大学);Shenzhen Research Institute of Big Data(深圳大数据研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));National University of Singapore(新加坡国立大学)
Point to the Hidden: Exposing Speech Audio Splicing via Signal Pointer Nets
Federal Criminal Police Office (BKA)(联邦刑事警察局(BKA));Friedrich-Alexander University of Erlangen-Nürnberg(埃尔朗根-纽伦堡弗里德里希-亚历山大大学);Julius-Maximilians University of Würzburg(维尔茨堡朱利叶斯-马克西米利安大学)
Jailbreak-AudioBench: In-Depth Evaluation and Analysis of Jailbreak Threats for Large Audio Language Models
Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));University of Oxford(牛津大学);Xi’an Jiaotong University(西安交通大学);Hong Kong University of Science and Technology(香港科技大学);Northeastern University(东北大学);Beijing University of Technology(北京理工大学)
SongEditor: Adapting Zero-Shot Song Generation Language Model as a Multi-Task Editor
The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)(香港中文大学(深圳));Tencent AI Lab(腾讯AI实验室);Shenzhen Research Institute of Big Data(深圳大数据研究院);Tsinghua University(清华大学);Shanghai Jiao Tong University(上海交通大学);Nanjing University(南京大学)