The generation of natural and high-quality speech from text is a challenging problem in the field of natural language processing. In addition to speech generation, speech editing is also a crucial task, which requires the seamless and unnoticeable integration of edited speech into synthesized speech. We propose a novel approach to speech editing by leveraging a pre-trained text-to-speech (TTS) model, such as FastSpeech 2, and incorporating a double attention block network on top of it to automatically merge the synthesized mel-spectrogram with the mel-spectrogram of the edited text. We refer to this model as AttentionStitch, as it harnesses attention to stitch audio samples together. We evaluate the proposed AttentionStitch model against state-of-the-art baselines on both single and multi-speaker datasets, namely LJSpeech and VCTK. We demonstrate its superior performance through an objective and a subjective evaluation test involving 15 human participants. AttentionStitch is capable of producing high-quality speech, even for words not seen during training, while operating automatically without the need for human intervention. Moreover, AttentionStitch is fast during both training and inference and is able to generate human-sounding edited speech.
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
SeamlessEdit: Background Noise Aware Zero-Shot Speech Editing with in-Context Enhancement
GICE, National Taiwan University, Taipei, Taiwan(国家台湾大学资讯工程研究所,台北,台湾);AI Research Center, Inventec Corporation, Taipei, Taiwan(Inventec公司人工智能研究中心,台北,台湾)
Speech Driven Video Editing via an Audio-Conditioned Diffusion Model
University of Galway(高威大学);University of Wrocław(弗罗茨瓦夫大学);Wrocław University of Science and Technology(弗罗茨瓦夫理工大学);Tooploox(Tooploox公司);Trinity College Dublin(都柏林圣三一学院)
The State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Department of Automation, Tsinghua University(清华大学自动化系);BNRist, Tsinghua University(清华大学BNRist)
FluentEditor: Text-based Speech Editing by Considering Acoustic and Prosody Consistency
Inner Mongolia University(内蒙古大学);Zhejiang University(浙江大学);Shenzhen Research Institute of Big Data(深圳大数据研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));National University of Singapore(新加坡国立大学)
Fine-Grained Quantitative Emotion Editing for Speech Generation
School of Data Science(数据科学学院);Shenzhen Research Institute of Big Data(深圳大数据研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Alibaba Group(阿里巴巴集团)
Towards Emotionally Consistent Text-Based Speech Editing: Introducing EmoCorrector and The ECD-TSE Dataset
Johns Hopkins University(约翰霍普金斯大学);Carnegie Mellon University(卡内基梅隆大学);The Chinese University of Hong Kong(香港中文大学);National University of Singapore(新加坡国立大学);Inner Mongolia University(内蒙古大学)
Cross-Utterance Conditioned VAE for Speech Generation
the University of Manchester(曼彻斯特大学);ShanghaiTech University(上海科技大学);University of Cambridge(剑桥大学);Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学);University College London(伦敦大学学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));University of Shanghai for Science and Technology(上海理工大学)
UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion
The Chinese University of Hong Kong(香港中文大学);The Hong Kong Polytechnic University(香港理工大学);City University of Hong Kong(香港城市大学);The Hong Kong University of Science and Technology(香港科学与技术大学);Tsinghua University(清华大学);Huawei Research Hong Kong(华为香港研究)
UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding
MoE Key Lab of Artificial Intelligence, AI Institute X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University(上海交通大学电子信息与电气工程学院计算机科学与工程系人工智能教育部重点实验室X-LANCE实验室);Shenzhen Research Institute of Big Data(深圳市大数据研究院);AISpeech Ltd(思必驰科技股份有限公司)
National Engineering Research Center of Speech and Language Information Processing(国家语言与语音信息处理工程研究中心);University of Science and Technology of China(中国科学技术大学);iFLYTEK Research(讯飞研究院)
Benchmarking Evaluation Metrics for Code-Switching Automatic Speech Recognition
New York University Abu Dhabi(纽约大学阿布扎比分校);University of Stuttgart(斯图加特大学);Johns Hopkins University(约翰斯·霍普金斯大学);Stanford University(斯坦福大学);Qatar Computing Research Institute(卡塔尔计算研究所);Microsoft Research India(微软印度研究院)