This paper presents a new neural speech compression method that is practical in the sense that it operates at low bitrate, introduces a low latency, is compatible in computational complexity with current mobile devices, and provides a subjective quality that is comparable to that of standard mobile-telephony codecs. Other recently proposed neural vocoders also have the ability to operate at low bitrate. However, they do not produce the same level of subjective quality as standard codecs. On the other hand, standard codecs rely on objective and short-term metrics such as the segmental signal-to-noise ratio that correlate only weakly with perception. Furthermore, standard codecs are less efficient than unsupervised neural networks at capturing speech attributes, especially long-term ones. The proposed method combines a cognitive-coding encoder that extracts an interpretable unsupervised hierarchical representation with a multi stage decoder that has a GAN-based architecture. We observe that this method is very robust to the quantization of representation features. An AB test was conducted on a subset of the Harvard sentences that are commonly used to evaluate standard mobile-telephony codecs. The results show that the proposed method outperforms the standard AMR-WB codec in terms of delay, bitrate and subjective quality.
National University of Defense Technology（国防科技大学）；Army Engineering University of PLA（中国人民解放军陆军工程大学）
Inspired by the success of deep neural networks (DNNs) in speech processing, this paper presents Deep Vocoder, a direct end-to-end low bit rate speech compression method with deep autoencoder (DAE). In Deep Vocoder, DAE is used for extracting the latent representing features (LRFs) of speech, which are then efficiently quantized by an analysis-by-synthesis vector quantization (AbS VQ) method. AbS VQ aims to minimize the perceptual spectral reconstruction distortion rather than the distortion of LRFs vector itself. Also, a suboptimal codebook searching technique is proposed to further reduce th…
There are many deterministic mathematical operations (e.g. compression, clipping, downsampling) that degrade speech quality considerably. In this paper we introduce a neural network architecture, based on a modification of the DiffWave model, that aims to restore the original speech signal. DiffWave, a recently published diffusion-based vocoder, has shown state-of-the-art synthesized speech quality and relatively shorter waveform generation times, with only a small set of parameters. We replace the mel-spectrum upsampler in DiffWave with a deep CNN upsampler, which is trained to alter the degr…
A Neural Speech Codec for Noise Robust Speech Coding
Brain-Inspired Application Technology Center (BATC), School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University（脑启发应用技术中心（BATC）、电子信息与电气工程学院、上海交通大学）；Hangzhou Dianzi University（杭州电子科技大学）；Shanghai Aero Measurement Controlling Research Institute（上海航空测量控制研究所）
Modern compression algorithms are often the result of laborious domain-specific research; industry standards such as MP3, JPEG, and AMR-WB took years to develop and were largely hand-designed. We present a deep neural network model which optimizes all the steps of a wideband speech coding pipeline (compression, quantization, entropy coding, and decompression) end-to-end directly from raw speech data -- no manual feature engineering necessary, and it trains in hours. In testing, our DNN-based coder performs on par with the AMR-WB standard at a variety of bitrates (~9kbps up to ~24kbps). It also…
HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling
The School of Automation Science and Engineering, Xi'an Jiaotong University, Xi'an, China（自动化科学与工程学院，西安交通大学）；Shanghai AI Laboratory, Shanghai, China（上海人工智能实验室）；SenseTime Research, Shanghai, China（商汤研究）；Shanghai Jiao Tong University, Shanghai, China（上海交通大学）；The Chinese University of Hong Kong, Hong Kong SAR, China（香港中文大学）
WaveNet is a state-of-the-art text-to-speech vocoder that remains challenging to deploy due to its autoregressive loop. In this work we focus on ways to accelerate the original WaveNet architecture directly, as opposed to modifying the architecture, such that the model can be deployed as part of a scalable text-to-speech system. We survey a wide variety of model compression techniques that are amenable to deployment on a range of hardware platforms. In particular, we compare different model sparsity methods and levels, and seven widely used precisions as targets for quantization; and are able …
South China University of Technology（华南理工大学）；UBTECH Robotics Corp（优必选机器人公司）；iFLYTEK Research（科大讯飞研究院）
Transformer has obtained promising results on cognitive speech signal processing field, which is of interest in various applications ranging from emotion to neurocognitive disorder analysis. However, most works treat speech signal as a whole, leading to the neglect of the pronunciation structure that is unique to speech and reflects the cognitive process. Meanwhile, Transformer has heavy computational burden due to its full attention operation. In this paper, a hierarchical efficient framework, called SpeechFormer, which considers the structural characteristics of speech, is proposed and can b…
VoxCog: Towards End-to-End Multilingual Cognitive Impairment Classification through Dialectal Knowledge
Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California（明斯希德电气与计算机工程系，南加州大学）；Dornsife College of Letters, Arts and Sciences, University of Southern California（多恩斯菲学院，南加州大学）
Nanjing University（南京大学）；Horizon Robotics（地平线机器人）；Nanjing Institute of Advanced Artificial Intelligence（南京前沿人工智能研究院）
The direct expansion of deep neural network (DNN) based wide-band speech enhancement (SE) to full-band processing faces the challenge of low frequency resolution in low frequency range, which would highly likely lead to deteriorated performance of the model. In this paper, we propose a learnable spectral compression mapping (SCM) to effectively compress the high frequency components so that they can be processed in a more efficient manner. By doing so, the model can pay more attention to low and middle frequency range, where most of the speech power is concentrated. Instead of suppressing nois…
Connected Speech-Based Cognitive Assessment in Chinese and English
Usher Institute, Medical School（医学院厄舍研究所）；School of Health in Social Science（社会科学健康学院）；School of Engineering, University of Edinburgh（爱丁堡大学工程学院）；Carnegie Mellon University（卡内基梅隆大学）；University of Delaware（特拉华大学）；Cardinal Tien Hospital（圣保禄医院）
Disentangling peripheral hearing loss from central and cognitive effects on speech intelligibility in older adults
Center for Innovative and Joint Research, Wakayama University（和歌山大学创新与联合研究中心）；Graduate School of Systems Engineering, Wakayama University（和歌山大学系统工学研究科）；School of Informatics and Engineering, the University of Electro-Communications（电气通信大学信息与工程学院）
It is widely accepted that information derived from analyzing speech (the acoustic signal) and language production (words and sentences) serves as a useful window into the health of an individual's cognitive ability. In fact, most neuropsychological testing batteries have a component related to speech and language where clinicians elicit speech from patients for subjective evaluation across a broad set of dimensions. With advances in speech signal processing and natural language processing, there has been recent interest in developing tools to detect more subtle changes in cognitive-linguistic…
Alzheimer's disease (AD) is a progressive neurodegenerative disease most often associated with memory deficits and cognitive decline. With the aging population, there has been much interest in automated methods for cognitive impairment detection. One approach that has attracted attention in recent years is AD detection through spontaneous speech. While the results are promising, it is not certain whether the learned speech features can be generalized across languages. To fill this gap, the ADReSS-M challenge was organized. This paper presents our submission to this ICASSP-2023 Signal Processin…
Deep speaker embeddings have been shown effective for assessing cognitive impairments aside from their original purpose of speaker verification. However, the research found that speaker embeddings encode speaker identity and an array of information, including speaker demographics, such as sex and age, and speech contents to an extent, which are known confounders in the assessment of cognitive impairments. In this paper, we hypothesize that content information separated from speaker identity using a framework for voice conversion is more effective for assessing cognitive impairments and train s…
Automatic speech recognition (ASR) in the cloud allows the use of larger models and more powerful multi-channel signal processing front-ends compared to on-device processing. However, it also adds an inherent latency due to the transmission of the audio signal, especially when transmitting multiple channels of a microphone array. One way to reduce the network bandwidth requirements is client-side compression with a lossy codec such as Opus. However, this compression can have a detrimental effect especially on multi-channel ASR front-ends, due to the distortion and loss of spatial information i…
We compare a series of time compression methods applied to normal and clear speech. First we evaluate a linear (uniform) method applied to these styles as well as to naturally-produced fast speech. We found, in line with the literature, that unprocessed fast speech was less intelligible than linearly compressed normal speech. Fast speech was also less intelligible than compressed clear speech but at the highest rate (three times faster than normal) the advantage of clear over fast speech was lost. To test whether this was due to shorter speech duration we evaluate, in our second experiments, a…
The Voice of Equity: A Systematic Evaluation of Bias Mitigation Techniques for Speech-Based Cognitive Impairment Detection Across Architectures and Demographics
Columbia University Irving Medical Center（哥伦比亚大学欧文医学中心）；Columbia University School of Nursing（哥伦比亚大学护理学院）；Columbia University（哥伦比亚大学）
Usher Institute, Edinburgh Medical School, The University of Edinburgh（爱丁堡大学爱丁堡医学院厄舍研究所）；Carnegie Mellon University（卡内基梅隆大学）
Building on the success of the ADReSS Challenge at Interspeech 2020, which attracted the participation of 34 teams from across the world, the ADReSSo Challenge targets three difficult automatic prediction problems of societal and medical relevance, namely: detection of Alzheimer's Dementia, inference of cognitive testing scores, and prediction of cognitive decline. This paper presents these prediction tasks in detail, describes the datasets used, and reports the results of the baseline classification and regression models we developed for each task. A combination of acoustic and linguistic fea…
We propose an approach for cognitive coding of speech by unsupervised extraction of contextual representations in two hierarchical levels of abstraction. Speech attributes such as phoneme identity that last one hundred milliseconds or less are captured in the lower level of abstraction, while speech attributes such as speaker identity and emotion that persist up to one second are captured in the higher level of abstraction. This decomposition is achieved by a two-stage neural network, with a lower and an upper stage operating at different time scales. Both stages are trained to predict the con…
Unfolding A Few Structures for The Many: Memory-Efficient Compression of Conformer and Speech Foundation Models
The Chinese University of Hong Kong（香港中文大学）；Chinese Academy of Sciences（中国科学院）；Tsinghua University（清华大学）；Institute of Software（软件研究所）；Department of Electronic Engineering（电子工程系）
On the compression of shallow non-causal ASR models using knowledge distillation and tied-and-reduced decoder for low-latency on-device speech recognition
Institute of Acoustics, Chinese Academy of Sciences（中国科学院声学研究所）；University of Chinese Academy of Sciences（中国科学院大学）；School of Science, Renmin University of China（中国人民大学理学院）；Xinjiang Technical Institute of Physics and Chemistry, Chinese Academy of Sciences（中国科学院新疆理化技术研究所）
The deep neural network (DNN) based speech enhancement approaches have achieved promising performance. However, the number of parameters involved in these methods is usually enormous for the real applications of speech enhancement on the device with the limited resources. This seriously restricts the applications. To deal with this issue, model compression techniques are being widely studied. In this paper, we propose a model compression method based on matrix product operators (MPO) to substantially reduce the number of parameters in DNN models for speech enhancement. In this method, the weig…
NTT Communication Science Laboratories, NTT Corporation（NTT公司NTT通信科学实验室）；University of Oldenburg（奥尔登堡大学）
The performance of speech enhancement algorithms in a multi-speaker scenario depends on correctly identifying the target speaker to be enhanced. Auditory attention decoding (AAD) methods allow to identify the target speaker which the listener is attending to from single-trial EEG recordings. Aiming at enhancing the target speaker and suppressing interfering speakers, reverberation and ambient noise, in this paper we propose a cognitive-driven multi-microphone speech enhancement system, which combines a neural-network-based mask estimator, weighted minimum power distortionless response convolut…
Explainable and Generalisable LLM-based Cognitive Decline Detection with Spontaneous Speech
Tsinghua University（清华大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Peking University Sixth Hospital（北京大学第六医院）；Peking University（北京大学）；OPPO Research Institute（OPPO研究院）；Peking University Third Hospital（北京大学第三医院）
This paper introduces a novel neural network-based speech coding system that can process noisy speech effectively. The proposed source-aware neural audio coding (SANAC) system harmonizes a deep autoencoder-based source separation model and a neural coding system so that it can explicitly perform source separation and coding in the latent space. An added benefit of this system is that the codec can allocate a different amount of bits to the underlying sources so that the more important source sounds better in the decoded signal. We target a new use case where the user on the receiver side cares…
Chair of Human-Machine Communication, Department of Electrical and Computer Engineering, Technical University of Munich（慕尼黑工业大学电气与计算机工程系人机通信教席）
Audio Adversarial Examples (AAE) represent specially created inputs meant to trick Automatic Speech Recognition (ASR) systems into misclassification. The present work proposes MP3 compression as a means to decrease the impact of Adversarial Noise (AN) in audio samples transcribed by ASR systems. To this end, we generated AAEs with the Fast Gradient Sign Method for an end-to-end, hybrid CTC-attention ASR system. Our method is then validated by two objective indicators: (1) Character Error Rates (CER) that measure the speech decoding performance of four ASR models trained on uncompressed, as wel…
Deep neural network based full-band speech enhancement systems face challenges of high demand of computational resources and imbalanced frequency distribution. In this paper, a light-weight full-band model is proposed with two dedicated strategies, i.e., a learnable spectral compression mapping for more effective high-band spectral information compression, and the utilization of the multi-head attention mechanism for more effective modeling of the global spectral pattern. Experiments validate the efficacy of the proposed strategies and show that the proposed model achieves competitive performa…
The smaller memory bandwidth in smart devices prompts development of smaller Automatic Speech Recognition (ASR) models. To obtain a smaller model, one can employ the model compression techniques. Knowledge distillation (KD) is a popular model compression approach that has shown to achieve smaller model size with relatively lesser degradation in the model performance. In this approach, knowledge is distilled from a trained large size teacher model to a smaller size student model. Also, the transducer based models have recently shown to perform well for on-device streaming ASR task, while the co…
National Taiwan University（台湾大学）；Academia Sinica（中央研究院）；Johns Hopkins University（约翰斯·霍普金斯大学）；City University of Hong Kong（香港城市大学）
Numerous compression and acceleration strategies have achieved outstanding results on classification tasks in various fields, such as computer vision and speech signal processing. Nevertheless, the same strategies have yielded ungratified performance on regression tasks because the nature between these and classification tasks differs. In this paper, a novel sign-exponent-only floating-point network (SEOFP-NET) technique is proposed to compress the model size and accelerate the inference time for speech enhancement, a regression task of speech signal processing. The proposed method compressed …
Beijing University of Posts and Telecommunications（北京邮电大学）；Peng Cheng Laboratory（鹏城实验室）
We propose a novel neural waveform compression method to catalyze emerging speech semantic communications. By introducing nonlinear transform and variational modeling, we effectively capture the dependencies within speech frames and estimate the probabilistic distribution of the speech feature more accurately, giving rise to better compression performance. In particular, the speech signals are analyzed and synthesized by a pair of nonlinear transforms, yielding latent features. An entropy model with hyperprior is built to capture the probabilistic distribution of latent features, followed with…
As a neurophysiological response to threat or adverse conditions, stress can affect cognition, emotion and behaviour with potentially detrimental effects on health in the case of sustained exposure. Since the affective content of speech is inherently modulated by an individual's physical and mental state, a substantial body of research has been devoted to the study of paralinguistic correlates of stress-inducing task load. Historically, voice stress analysis (VSA) has been conducted using conventional digital signal processing (DSP) techniques. Despite the development of modern methods based o…
