We construct audio adversarial examples on automatic Speech-To-Text systems . Given any audio waveform, we produce an another by overlaying an audio vocal mask generated from the original audio. We apply our audio adversarial attack to five SOTA STT systems: DeepSpeech, Julius, Kaldi, wav2letter@anywhere and CMUSphinx. In addition, we engaged human annotators to transcribe the adversarial audio. Our experiments show that these adversarial examples fool State-Of-The-Art Speech-To-Text systems, yet humans are able to consistently pick out the speech. The feasibility of this attack introduces a new domain to study machine and human perception of speech.
Multimodal Laryngoscopic Video Analysis for Assisted Diagnosis of Vocal Fold Paralysis
School of Computer Science（计算机科学学院）；Suzhou Municipal Key Laboratory of Multimodal Intelligent Systems（多模态智能系统苏州市重点实验室）；Sun Yat-sen Memorial Hospital of Sun Yat-sen University（中山大学孙逸仙纪念医院）；School of Artificial Intelligent（人工智能学院）；Department of Otorhinolaryngology（耳鼻喉科部门）
Automatic lyrics to polyphonic audio alignment is a challenging task not only because the vocals are corrupted by background music, but also there is a lack of annotated polyphonic corpus for effective acoustic modeling. In this work, we propose (1) using additional speech and music-informed features and (2) adapting the acoustic models trained on a large amount of solo singing vocals towards polyphonic music using a small amount of in-domain data. Incorporating additional information such as voicing and auditory features together with conventional acoustic features aims to bring robustness ag…
In this paper, we consider the task of digitally voicing silent speech, where silently mouthed words are converted to audible speech based on electromyography (EMG) sensor measurements that capture muscle impulses. While prior work has focused on training speech synthesis models from EMG collected during vocalized speech, we are the first to train from EMG collected during silently articulated speech. We introduce a method of training on silent EMG by transferring audio targets from vocalized to silent signals. Our method greatly improves intelligibility of audio generated from silent EMG comp…
University of Rochester（罗切斯特大学）；ByteDance Inc.（字节跳动公司）
Separating a song into vocal and accompaniment components is an active research topic, and recent years witnessed an increased performance from supervised training using deep learning techniques. We propose to apply the visual information corresponding to the singers' vocal activities to further improve the quality of the separated vocal signals. The video frontend model takes the input of mouth movement and fuses it into the feature embeddings of an audio-based separation framework. To facilitate the network to learn audiovisual correlation of singing activities, we add extra vocal signals ir…
Learning Marmoset Vocal Patterns with a Masked Autoencoder for Robust Call Segmentation, Classification, and Caller Identification
RIKEN Center for Advanced Intelligence Project (AIP), Japan（日本先进人工智能项目研究所）；Keio University, Japan（庆应大学）；Nara Institute of Science and Technology, Japan（奈良科学技術大学）；The Chinese University of Hong Kong, Shenzhen, China（香港中文大学（深圳））
VocalAgent: Large Language Models for Vocal Health Diagnostics with Safety-Aware Evaluation
Massachusetts Institute of Technology, USA（麻省理工学院, 美国）；Seoul National University Hospital, South Korea（首尔国立大学医院, 韩国）；Doctor Diary, South Korea（Doctor Diary, 韩国）；Google Research, USA（谷歌研究, 美国）；Independent Researcher（独立研究者）
Stanford University（斯坦福大学）；Stanford School of Medicine（斯坦福大学医学院）
In this paper, we propose a method for removing linguistic information from speech for the purpose of isolating paralinguistic indicators of affect. The immediate utility of this method lies in clinical tests of sensitivity to vocal affect that are not confounded by language, which is impaired in a variety of clinical populations. The method is based on simultaneous recordings of speech audio and electroglottographic (EGG) signals. The speech audio signal is used to estimate the average vocal tract filter response and amplitude envelop. The EGG signal supplies a direct correlate of voice sourc…
We present SingSong, a system that generates instrumental music to accompany input vocals, potentially offering musicians and non-musicians alike an intuitive new way to create music featuring their own voice. To accomplish this, we build on recent developments in musical source separation and audio generation. Specifically, we apply a state-of-the-art source separation algorithm to a large corpus of music audio to produce aligned pairs of vocals and instrumental sources. Then, we adapt AudioLM (Borsos et al., 2022) -- a state-of-the-art approach for unconditional audio generation -- to be sui…
In this paper, we address the problem of lip-voice synchronisation in videos containing human face and voice. Our approach is based on determining if the lips motion and the voice in a video are synchronised or not, depending on their audio-visual correspondence score. We propose an audio-visual cross-modal transformer-based model that outperforms several baseline models in the audio-visual synchronisation task on the standard lip-reading speech benchmark dataset LRS2. While the existing methods focus mainly on lip synchronisation in speech videos, we also consider the special case of the sing…
Universitat Pompeu Fabra（庞培法布拉大学）；KTH Royal Institute of Technology（瑞典皇家理工学院）
The goal of this study is the automatic detection of onsets of the singing voice in polyphonic audio recordings. Starting with a hypothesis that the knowledge of the current position in a metrical cycle (i.e. metrical accent) can improve the accuracy of vocal note onset detection, we propose a novel probabilistic model to jointly track beats and vocal note onsets. The proposed model extends a state of the art model for beat and meter tracking, in which a-priori probability of a note at a specific metrical accent interacts with the probability of observing a vocal note onset. We carry out an ev…
EMVD dataset: a dataset of extreme vocal distortion techniques used in heavy metal
Nantes Université（南特大学）；École Centrale Nantes（南特中央理工学院）；CNRS（法国国家科学研究中心）；LS2N（南特数字科学实验室）；Institut Acte（阿克特研究所）；ENS Louis-Lumière（路易·卢米埃尔高等师范学校）；IRIT（图灵与信息技术研究所）；Université de Toulouse（图卢兹大学）；UT3（图卢兹第三大学）
We describe our approach for the generative emotional vocal burst task (ExVo Generate) of the ICML Expressive Vocalizations Competition. We train a conditional StyleGAN2 architecture on mel-spectrograms of preprocessed versions of the audio samples. The mel-spectrograms generated by the model are then inverted back to the audio domain. As a result, our generated samples substantially improve upon the baseline provided by the competition from a qualitative and quantitative perspective for all emotions. More precisely, even for our worst-performing emotion (awe), we obtain an FAD of 1.76 compare…
The potential use of non-linear speech features has not been investigated for music analysis although other commonly used speech features like Mel Frequency Ceptral Coefficients (MFCC) and pitch have been used extensively. In this paper, we assume an audio signal to be a sum of modulated sinusoidal and then use the energy separation algorithm to decompose the audio into amplitude and frequency modulation components using the non-linear Teager-Kaiser energy operator. We first identify the distribution of these non-linear features for music only and voice only segments in the audio signal in dif…
A large-scale and PCR-referenced vocal audio dataset for COVID-19
London Centre for Nanotechnology, University College London（伦敦纳米技术中心，伦敦大学学院）；Division of Medicine, University College London（医学系，伦敦大学学院）；King's College London, UK（伦敦国王学院）；The Alan Turing Institute, UK（阿兰·图灵研究所）；Imperial College London, UK（帝国理工学院）；UK Health Security Agency, UK（英国卫生安全局）；Institute of Health Informatics, University College London, UK（健康信息学研究所，伦敦大学学院）；Centre for Stress and Age-Related Disease, School of Applied Sciences, University of Brighton, UK（布莱顿大学应用科学学院压力与年龄相关疾病中心）；University of Oxford, UK（牛津大学）；University of Surrey, UK（萨里大学）；University of Lancaster, UK（兰卡斯特大学）；University of Cambridge, UK（剑桥大学）；University of Augsburg, Germany（奥格斯堡大学）；University of Nottingham, UK（诺丁汉大学）
In the field of audio signal processing research, source separation has been a popular research topic for a long time and the recent adoption of the deep neural networks have shown a significant improvement in performance. The improvement vitalizes the industry to productize audio deep learning based products and services including Karaoke in the music streaming apps and dialogue enhancement in the UHDTV. For these early markets, we defined a set of design principles of the vocal and accompaniment separation model in terms of robustness, quality, and cost. In this paper, we introduce GSEP (Gau…
Automatic speaker recognition algorithms typically characterize speech audio using short-term spectral features that encode the physiological and anatomical aspects of speech production. Such algorithms do not fully capitalize on speaker-dependent characteristics present in behavioral speech features. In this work, we propose a prosody encoding network called DeepTalk for extracting vocal style features directly from raw audio data. The DeepTalk method outperforms several state-of-the-art speaker recognition systems across multiple challenging datasets. The speaker recognition performance is f…
We present a neural text-to-speech (TTS) method that models natural vocal effort variation to improve the intelligibility of synthetic speech in the presence of noise. The method consists of first measuring the spectral tilt of unlabeled conventional speech data, and then conditioning a neural TTS model with normalized spectral tilt among other prosodic factors. Changing the spectral tilt parameter and keeping other prosodic factors unchanged enables effective vocal effort control at synthesis time independent of other prosodic factors. By extrapolation of the spectral tilt values beyond what …
University of British Columbia（英属哥伦比亚大学）；University of Alberta（阿尔伯塔大学）；Aalto University（阿尔托大学）
A state-of-the-art 1D acoustic synthesizer has been previously developed, and coupled to speaker-specific biomechanical models of oropharynx in ArtiSynth. As expected, the formant frequencies of the synthesized vowel sounds were shown to be different from those of the recorded audio. Such discrepancy was hypothesized to be due to the simplified geometry of the vocal tract model as well as the one dimensional implementation of Navier-Stokes equations. In this paper, we calculate Helmholtz resonances of our vocal tract geometries using 3D finite element method (FEM), and compare them with the fo…
The tabla is a unique percussion instrument due to the combined harmonic and percussive nature of its timbre, and the contrasting harmonic frequency ranges of its two drums. This allows a tabla player to uniquely emphasize parts of the rhythmic cycle (theka) in order to mark the salient positions. An analysis of the loudness dynamics and timing deviations at various cycle positions is an important part of musicological studies on the expressivity in tabla accompaniment. To achieve this at a corpus-level, and not restrict it to the few recordings that manual annotation can afford, it is helpful…
RapVerse: Coherent Vocals and Whole-Body Motions Generations from Text
UMass Amherst（马萨诸塞大学阿默斯特分校）；Wuhan University（武汉大学）；UC San Diego（加州大学圣地亚哥分校）；Tsinghua University（清华大学）；MIT-IBM Watson AI Lab（麻省理工-IBM沃森人工智能实验室）；Dolby Laboratories（杜比实验室）
Drum-to-Vocal Percussion Sound Conversion and Its Evaluation Methodology
Graduate School of Information Science and Technology, University of Tokyo（东京大学信息科学与技术研究生院）；National Institute of Advanced Industrial Science and Technology (AIST)（国家先进工业科学与技术研究院）；Faculty of Science and Technology, Keio University（庆应大学理工学院）
LeVo: High-Quality Song Generation with Multi-Preference Alignment
Shenzhen International Graduate School, Tsinghua University, Shenzhen（清华大学深圳国际研究生院）；Tencent AI Lab（腾讯AI实验室）；Wuhan University（武汉大学）；The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（香港中文大学（深圳））；X-LANCE Lab, Shanghai Jiao Tong University, Shanghai（上海交通大学X-LANCE实验室）；School of Intelligence Science and Technology, Nanjing University, Suzhou, China（南京大学智能科学与技术学院）
Recognizing human non-speech vocalizations is an important task and has broad applications such as automatic sound transcription and health condition monitoring. However, existing datasets have a relatively small number of vocal sound samples or noisy labels. As a consequence, state-of-the-art audio event classification models may not perform well in detecting human vocal sounds. To support research on building robust and accurate vocal sound recognition, we have created a VocalSound dataset consisting of over 21,000 crowdsourced recordings of laughter, sighs, coughs, throat clearing, sneezes,…
