Model size and complexity remain the biggest challenges in the deployment of speech enhancement and separation systems on low-resource devices such as earphones and hearing aids. Although methods such as compression, distillation and quantization can be applied to large models, they often come with a cost on the model performance. In this paper, we provide a simple model design paradigm that explicitly designs ultra-lightweight models without sacrificing the performance. Motivated by the sub-band frequency-LSTM (F-LSTM) architectures, we introduce the group communication (GroupComm), where a feature vector is split into smaller groups and a small processing block is used to perform inter-group communication. Unlike standard F-LSTM models where the sub-band outputs are concatenated, an ultra-small module is applied on all the groups in parallel, which allows a significant decrease on the model size. Experiment results show that comparing with a strong baseline model which is already lightweight, GroupComm can achieve on par performance with 35.6 times fewer parameters and 2.3 times fewer operations.
A Fast and Lightweight Model for Causal Audio-Visual Speech Separation
School of Computer Technology and Application（计算机技术与应用学院）；Intelligent Computing and Application Laboratory of Qinghai Province（青海省智能计算与应用实验室）；Department of Computer Science and Technology（计算机科学与技术系）；Institute for AI（人工智能研究院）；Tsinghua Laboratory of Brain and Intelligence (THBI)（清华大学脑与智能实验室）；IDG/McGovern Institute for Brain Research（IDG/麦克戈维脑研究学院）；Tsinghua University（清华大学）；Chinese Institute for Brain Research (CIBR)（中国脑科学研究院）
Harbin Institute of Technology（哈尔滨工业大学）；Microsoft Corporation（微软公司）
Transformer has been successfully applied to speech separation recently with its strong long-dependency modeling capacity using a self-attention mechanism. However, Transformer tends to have heavy run-time costs due to the deep encoder layers, which hinders its deployment on edge devices. A small Transformer model with fewer encoder layers is preferred for computational efficiency, but it is prone to performance degradation. In this paper, an ultra fast speech separation Transformer model is proposed to achieve both better performance and efficiency with teacher student learning (T-S learning)…
University of Szeged（塞格德大学）；MTA-SZTE Research Group on Artificial Intelligence（匈牙利科学院-塞格德大学人工智能研究组）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）
Articulatory-to-acoustic mapping seeks to reconstruct speech from a recording of the articulatory movements, for example, an ultrasound video. Just like speech signals, these recordings represent not only the linguistic content, but are also highly specific to the actual speaker. Hence, due to the lack of multi-speaker data sets, researchers have so far concentrated on speaker-dependent modeling. Here, we present multi-speaker experiments using the recently published TaL80 corpus. To model speaker characteristics, we adjusted the x-vector framework popular in speech processing to operate with …
Voice-activated systems are integrated into a variety of desktop, mobile, and Internet-of-Things (IoT) devices. However, voice spoofing attacks, such as impersonation and replay attacks, in which malicious attackers synthesize the voice of a victim or simply replay it, have brought growing security concerns. Existing speaker verification techniques distinguish individual speakers via the spectrographic features extracted from an audible frequency range of voice commands. However, they often have high error rates and/or long delays. In this paper, we explore a new direction of human voice resea…
Automatic speech synthesis is a challenging task that is becoming increasingly important as edge devices begin to interact with users through speech. Typical text-to-speech pipelines include a vocoder, which translates intermediate audio representations into an audio waveform. Most existing vocoders are difficult to parallelize since each generated sample is conditioned on previous samples. WaveGlow is a flow-based feed-forward alternative to these auto-regressive models (Prenger et al., 2019). However, while WaveGlow can be easily parallelized, the model is too expensive for real-time speech …
Self-supervised learning method that provides generalized speech representations has recently received increasing attention. Wav2vec 2.0 is the most famous example, showing remarkable performance in numerous downstream speech processing tasks. Despite its success, it is challenging to use it directly for wake-up word detection on mobile devices due to its expensive computational cost. In this work, we propose LiteFEW, a lightweight feature encoder for wake-up word detection that preserves the inherent ability of wav2vec 2.0 with a minimum scale. In the method, the knowledge of the pre-trained …
FLY-TTS: Fast, Lightweight and High-Quality End-to-End Text-to-Speech Synthesis
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与工程学院）；College of Information Science（信息学院）；Electronic Engineering, Zhejiang University（浙江大学电子工程学院）
This paper introduces a new method referred to as KISS-GEV (for Keep It Super Simple Generalized eigenvalue) beamforming. While GEV beamforming usually relies on deep neural network for estimating target and noise time-frequency masks, this method uses a signal processing approach based on the direction of arrival (DoA) of the target. This considerably reduces the amount of computations involved at test time, and works for speech enhancement in unseen conditions as there is no need to train a neural network with noisy speech. The proposed method can also be used to separate speech from a mixtu…
THLNet: two-stage heterogeneous lightweight network for monaural speech enhancement
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Institute of Acoustics, CAS（中国科学院声学研究所）
ABSE-NET: A Lightweight Neural Model for Active Binaural Speech Enhancement in Open-Fit Hearing Aids
College of Computer Science, Inner Mongolia University（内蒙古大学计算机学院）；College of Electronic and Information Engineering, Inner Mongolia University（内蒙古大学电子与信息工程学院）
Dual-View Predictive Diffusion: Lightweight Speech Enhancement via Spectrogram-Image Synergy
School of Cyberspace Science and Technology, Beijing Institute of Technology, Beijing 100081, China（空间科学与技术学院，北京理工大学）；Department of Computer Science and Technology, Institute for AI, BNRist, Tsinghua University, Beijing 100084, China（计算机科学与技术系，人工智能研究院，清华大学）；School of Cyberspace Science and Technology Sun Yat-sen University, Guangzhou 510006, China（空间科学与技术学院，中山大学）
Dense-TSNet: Dense Connected Two-Stage Structure for Ultra-Lightweight Speech Enhancement
School of Electronic Information Sichuan University Chengdu, China；School of Communications；Information Engineering Chongqing University of Posts；College of Intelligence；Computing Tianjin University Tianjin, China junyu \；School of Medicine Tohoku University Sendai, Japan
We propose a lightweight end-to-end text-to-speech model using multi-band generation and inverse short-time Fourier transform. Our model is based on VITS, a high-quality end-to-end text-to-speech model, but adopts two changes for more efficient inference: 1) the most computationally expensive component is partially replaced with a simple inverse short-time Fourier transform, and 2) multi-band generation, with fixed or trainable synthesis filters, is used to generate waveforms. Unlike conventional lightweight models, which employ optimization or knowledge distillation separately to train two ca…
A Lightweight Fourier-based Network for Binaural Speech Enhancement with Spatial Cue Preservation
Shanghai Institute of Artificial Intelligence for Education, East China Normal University, China（上海人工智能教育研究院，华东师范大学，中国）；Boin Hearing Technology (Shanghai) Co., LTD, China（Boin听力技术（上海）有限公司，中国）；School of Computer Science and Technology, East China Normal University, China（计算机科学与技术学院，华东师范大学，中国）
Many end-to-end Automatic Speech Recognition (ASR) systems still rely on pre-processed frequency-domain features that are handcrafted to emulate the human hearing. Our work is motivated by recent advances in integrated learnable feature extraction. For this, we propose Lightweight Sinc-Convolutions (LSC) that integrate Sinc-convolutions with depthwise convolutions as a low-parameter machine-learnable feature extraction for end-to-end ASR systems. We integrated LSC into the hybrid CTC/attention architecture for evaluation. The resulting end-to-end model shows smooth convergence behaviour that i…
Indiana University（印第安纳大学）；Electronics and Telecommunications Research Institute（电子与电信研究院）
Speech codecs learn compact representations of speech signals to facilitate data transmission. Many recent deep neural network (DNN) based end-to-end speech codecs achieve low bitrates and high perceptual quality at the cost of model complexity. We propose a cross-module residual learning (CMRL) pipeline as a module carrier with each module reconstructing the residual from its preceding modules. CMRL differs from other DNN-based speech codecs, in that rather than modeling speech compression problem in a single large neural network, it optimizes a series of less-complicated modules in a two-pha…
University of Science and Technology of China（中国科学技术大学）；Microsoft Research Asia（微软亚洲研究院）；Microsoft Azure Speech（微软Azure语音）
Text to speech (TTS) has been broadly used to synthesize natural and intelligible speech in different scenarios. Deploying TTS in various end devices such as mobile phones or embedded devices requires extremely small memory usage and inference latency. While non-autoregressive TTS models such as FastSpeech have achieved significantly faster inference speed than autoregressive models, their model size and inference latency are still large for the deployment in resource constrained devices. In this paper, we propose LightSpeech, which leverages neural architecture search~(NAS) to automatically d…
NTT Communication Science Laboratories, NTT Corporation（NTT 公司 NTT 通信科学实验室）
In speech synthesis, a generative adversarial network (GAN), training a generator (speech synthesizer) and a discriminator in a min-max game, is widely used to improve speech quality. An ensemble of discriminators is commonly used in recent neural vocoders (e.g., HiFi-GAN) and end-to-end text-to-speech (TTS) systems (e.g., VITS) to scrutinize waveforms from multiple perspectives. Such discriminators allow synthesized speech to adequately approach real speech; however, they require an increase in the model size and computation time according to the increase in the number of discriminators. Alte…
Detecting emotions directly from a speech signal plays an important role in effective human-computer interactions. Existing speech emotion recognition models require massive computational and storage resources, making them hard to implement concurrently with other machine-interactive tasks in embedded systems. In this paper, we propose an efficient and lightweight fully convolutional neural network for speech emotion recognition in systems with limited hardware resources. In the proposed FCNN model, various feature maps are extracted via three parallel paths with different filter sizes. This h…
We introduce VANI, a very lightweight multi-lingual accent controllable speech synthesis system. Our model builds upon disentanglement strategies proposed in RADMMM and supports explicit control of accent, language, speaker and fine-grained $F_0$ and energy features for speech synthesis. We utilize the Indic languages dataset, released for LIMMITS 2023 as part of ICASSP Signal Processing Grand Challenge, to synthesize speech in 3 different languages. Our model supports transferring the language of a speaker while retaining their voice and the native accent of the target language. We utilize th…
We present a lightweight adaptable neural TTS system with high quality output. The system is composed of three separate neural network blocks: prosody prediction, acoustic feature prediction and Linear Prediction Coding Net as a neural vocoder. This system can synthesize speech with close to natural quality while running 3 times faster than real-time on a standard CPU. The modular setup of the system allows for simple adaptation to new voices with a small amount of data. We first demonstrate the ability of the system to produce high quality speech when trained on large, high quality datasets. …
National Taiwan University（国立台湾大学）；Intelligo Technology Inc.（Intelligo科技公司）
Personalized TTS is an exciting and highly desired application that allows users to train their TTS voice using only a few recordings. However, TTS training typically requires many hours of recording and a large model, making it unsuitable for deployment on mobile devices. To overcome this limitation, related works typically require fine-tuning a pre-trained TTS model to preserve its ability to generate high-quality audio samples while adapting to the target speaker's voice. This process is commonly referred to as ``voice cloning.'' Although related works have achieved significant success in c…
AS-ASR: A Lightweight Framework for Aphasia-Specific Automatic Speech Recognition
Department of Microelectronics, Delft University of Technology, The Netherlands（代尔夫特理工大学微电子系）；Leiden Institute of Advanced Computer Science (LIACS), Leiden University, The Netherlands（莱顿大学先进计算机科学研究所）；Sanford Health, Minnesota, United States（桑福医疗, 明尼苏达州, 美国）
Re-Parameterization of Lightweight Transformer for On-Device Speech Emotion Recognition
College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机与电子工程学院）；Shenzhen Research Institute, Hunan University（湖南大学深圳研究院）；University of Cambridge（剑桥大学）
