Recognizing human non-speech vocalizations is an important task and has broad applications such as automatic sound transcription and health condition monitoring. However, existing datasets have a relatively small number of vocal sound samples or noisy labels. As a consequence, state-of-the-art audio event classification models may not perform well in detecting human vocal sounds. To support research on building robust and accurate vocal sound recognition, we have created a VocalSound dataset consisting of over 21,000 crowdsourced recordings of laughter, sighs, coughs, throat clearing, sneezes, and sniffs from 3,365 unique subjects. Experiments show that the vocal sound recognition performance of a model can be significantly improved by 41.9% by adding VocalSound dataset to an existing dataset as training material. In addition, different from previous datasets, the VocalSound dataset contains meta information such as speaker age, gender, native language, country, and health condition.
Phonation, or the vibration of the vocal folds, is the primary source of vocalization in the production of voiced sounds by humans. It is a complex bio-mechanical process that is highly sensitive to changes in the speaker's respiratory parameters. Since most symptomatic cases of COVID-19 present with moderate to severe impairment of respiratory functions, we hypothesize that signatures of COVID-19 may be observable by examining the vibrations of the vocal folds. Our goal is to validate this hypothesis, and to quantitatively characterize the changes observed to enable the detection of COVID-19 …
Does My Dog ''Speak'' Like Me? The Acoustic Correlation between Pet Dogs and Their Human Owners
Institute for Clarity in Documentation（文档清晰度研究所）；Inria Paris-Rocquencourt（法国国家信息与自动化研究所巴黎-罗克库尔分部）；Rajiv Gandhi University（拉吉夫·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕尔默研究实验室）
University of Augsburg（奥格斯堡大学）；Imperial College London（帝国理工学院）
The field of sound healing includes ancient practices coming from a broad range of cultures. Across such practices there is a variety of acoustic instrumentation utilised. Practitioners suggest that sound has the ability to target both mental and even physical health issues, e.g., chronic-stress, or joint-pain. Instruments including the Tibetan singing bowl and vocal chanting, are still widely used today. With the noise-floor of modern urban soundscapes continually increasing and known to impact wellbeing, methods to improve this are needed. With that in mind, this study presents the Acoustic …
Searching sounds by text labels is often difficult, as text descriptions cannot describe the audio content in detail. Query by vocal imitation bridges such gap and provides a novel way to sound search. Several algorithms for sound search by vocal imitation have been proposed and evaluated in a simulation environment, however, they have not been deployed into a real search engine nor evaluated by real users. This pilot work conducts a subjective study to compare these two approaches to sound search, and tries to answer the question of which approach works better for what kinds of sounds. To do …
University of Palermo（帕勒莫大学）；Department of Mathematics and Computer Science（数学与计算机科学系）
Concepts and formalism from acoustics are often used to exemplify quantum mechanics. Conversely, quantum mechanics could be used to achieve a new perspective on acoustics, as shown by Gabor studies. Here, we focus in particular on the study of human voice, considered as a probe to investigate the world of sounds. We present a theoretical framework that is based on observables of vocal production, and on some measurement apparati that can be used both for analysis and synthesis. In analogy to the description of spin states of a particle, the quantum-mechanical formalism is used to describe the …
Bird sound data collected with unattended microphones for automatic surveys, or mobile devices for citizen science, typically contain multiple simultaneously vocalizing birds of different species. However, few works have considered the multi-label structure in birdsong. We propose to use an ensemble of classifier chains combined with a histogram-of-segments representation for multi-label classification of birdsong. The proposed method is compared with binary relevance and three multi-instance multi-label learning (MIML) algorithms from prior work (which focus more on structure in the sound, an…
Institut Systématique, Évolution, Biodiversité (ISYEB), Muséum national d’Histoire naturelle, CNRS, Sorbonne Université, EPHE, Université des Antilles（系统学、进化与生物多样性研究所（ISYEB），法国国家自然历史博物馆，法国国家科学研究中心，索邦大学，巴黎高等研究实践学院，安第列斯大学）
Open audio databases such as Xeno-Canto are widely used to build datasets to explore bird song repertoire or to train models for automatic bird sound classification by deep learning algorithms. However, such databases suffer from the fact that bird sounds are weakly labelled: a species name is attributed to each audio recording without timestamps that provide the temporal localization of the bird song of interest. Manual annotations can solve this issue, but they are time consuming, expert-dependent, and cannot run on large datasets. Another solution consists in using a labelling function that…
The expressive nature of the voice provides a powerful medium for communicating sonic ideas, motivating recent research on methods for query by vocalisation. Meanwhile, deep learning methods have demonstrated state-of-the-art results for matching vocal imitations to imitated sounds, yet little is known about how well learned features represent the perceptual similarity between vocalisations and queried sounds. In this paper, we address this question using similarity ratings between vocal imitations and imitated drum sounds. We use a linear mixed effect regression model to show how features lea…
Drum-to-Vocal Percussion Sound Conversion and Its Evaluation Methodology
Graduate School of Information Science and Technology, University of Tokyo（东京大学信息科学与技术研究生院）；National Institute of Advanced Industrial Science and Technology (AIST)（国家先进工业科学与技术研究院）；Faculty of Science and Technology, Keio University（庆应大学理工学院）
University of Augsburg（奥格斯堡大学）；Medical University of Graz（格拉茨医科大学）；Imperial College（帝国理工学院）
More than two years after its outbreak, the COVID-19 pandemic continues to plague medical systems around the world, putting a strain on scarce resources, and claiming human lives. From the very beginning, various AI-based COVID-19 detection and monitoring tools have been pursued in an attempt to stem the tide of infections through timely diagnosis. In particular, computer audition has been suggested as a non-invasive, cost-efficient, and eco-friendly alternative for detecting COVID-19 infections through vocal sounds. However, like all AI methods, also computer audition is heavily dependent on …
Cornell Lab of Ornithology, Cornell University（康奈尔大学康奈尔鸟类学实验室）；Music and Audio Research Laboratory, New York University（纽约大学音乐与音频研究实验室）；Center for Urban Science and Progress, New York University（纽约大学城市科学与进步中心）
Bioacoustic sensors, sometimes known as autonomous recording units (ARUs), can record sounds of wildlife over long periods of time in scalable and minimally invasive ways. Deriving per-species abundance estimates from these sensors requires detection, classification, and quantification of animal vocalizations as individual acoustic events. Yet, variability in ambient noise, both over time and across sensors, hinders the reliability of current automated systems for sound event detection (SED), such as convolutional neural networks (CNN) in the time-frequency domain. In this article, we develop,…
The ACII Affective Vocal Bursts (A-VB) competition introduces a new topic in affective computing, which is understanding emotional expression using the non-verbal sound of humans. We are familiar with emotion recognition via verbal vocal or facial expression. However, the vocal bursts such as laughs, cries, and signs, are not exploited even though they are very informative for behavior analysis. The A-VB competition comprises four tasks that explore non-verbal information in different spaces. This technical report describes the method and the result of SclabCNU Team for the tasks of the challe…
European Centre for Living Technology (ECLT)（欧洲生命技术中心）；Ca’ Foscari University of Venice（威尼斯大学）；University of Palermo（巴勒莫大学）
Humans have a privileged, embodied way to explore the world of sounds, through vocal imitation. The Quantum Vocal Theory of Sounds (QVTS) starts from the assumption that any sound can be expressed and described as the evolution of a superposition of vocal states, i.e., phonation, turbulence, and supraglottal myoelastic vibrations. The postulates of quantum mechanics, with the notions of observable, measurement, and time evolution of state, provide a model that can be used for sound processing, in both directions of analysis and synthesis. QVTS can give a quantum-theoretic explanation to some a…
Roli（Roli 公司）；Queen Mary University of London（伦敦玛丽女王大学）
The imitation of percussive sounds via the human voice is a natural and effective tool for communicating rhythmic ideas on the fly. Thus, the automatic retrieval of drum sounds using vocal percussion can help artists prototype drum patterns in a comfortable and quick way, smoothing the creative workflow as a result. Here we explore different strategies to perform this type of query, making use of both traditional machine learning algorithms and recent deep learning techniques. The main hyperparameters from the models involved are carefully selected by feeding performance metrics to a grid sear…
Imitating musical instruments with the human voice is an efficient way of communicating ideas between music producers, from sketching melody lines to clarifying desired sonorities. For this reason, there is an increasing interest in building applications that allow artists to efficiently pick target samples from big sound libraries just by imitating them vocally. In this study, we investigated the potential of conditional autoencoder models to learn informative features for Drum Sample Retrieval by Vocalisation (DSRV). We assessed the usefulness of their embeddings using four evaluation metric…
Ecole des Hautes Etudes en Sciences Sociales（巴黎高等社会科学研究院）；Centre National de la Recherche Scientifique（法国国家科学研究中心）；INRIA（法国国家信息与自动化研究所）；Ecole Normale Supérieure-Paris Sciences & Lettres University（巴黎文理研究大学高等师范学院）；Ecole Normale Supérieure（巴黎高等师范学院）；Institut national de la santé et de la recherche médicale（法国国家健康与医学研究院）；Institut Mondor de Recherche Biomédicale（蒙多生物医学研究所）；Neuratris（Neuratris研究所）；Université Paris-Est Créteil（巴黎东克雷泰伊大学）；Paris Sciences & Lettres University（巴黎文理研究大学）；Facebook AI Research（脸书人工智能研究院）
Deep Learning models have become potential candidates for auditory neuroscience research, thanks to their recent successes on a variety of auditory tasks. Yet, these models often lack interpretability to fully understand the exact computations that have been performed. Here, we proposed a parametrized neural network layer, that computes specific spectro-temporal modulations based on Gabor kernels (Learnable STRFs) and that is fully interpretable. We evaluated predictive capabilities of this layer on Speech Activity Detection, Speaker Verification, Urban Sound Classification and Zebra Finch Cal…
We introduce a novel approach to studying animal behaviour and the context in which it occurs, through the use of microphone backpacks carried on the backs of individual free-flying birds. These sensors are increasingly used by animal behaviour researchers to study individual vocalisations of freely behaving animals, even in the field. However such devices may record more than an animals vocal behaviour, and have the potential to be used for investigating specific activities (movement) and context (background) within which vocalisations occur. To facilitate this approach, we investigate the au…
The articulatory geometric configurations of the vocal tract and the acoustic properties of the resultant speech sound are considered to have a strong causal relationship. This paper aims at finding a joint latent representation between the articulatory and acoustic domain for vowel sounds via invertible neural network models, while simultaneously preserving the respective domain-specific features. Our model utilizes a convolutional autoencoder architecture and normalizing flow-based models to allow both forward and inverse mappings in a semi-supervised manner, between the mid-sagittal vocal t…
This paper examines the speaker identification potential of breath sounds in continuous speech. Speech is largely produced during exhalation. In order to replenish air in the lungs, speakers must periodically inhale. When inhalation occurs in the midst of continuous speech, it is generally through the mouth. Intra-speech breathing behavior has been the subject of much study, including the patterns, cadence, and variations in energy levels. However, an often ignored characteristic is the {\em sound} produced during the inhalation phase of this cycle. Intra-speech inhalation is rapid and energet…
Many approaches have been used in bird species classification from their sound in order to provide labels for the whole of a recording. However, a more precise classification of each bird vocalization would be of great importance to the use and management of sound archives and bird monitoring. In this work, we introduce a technique that using a two step process can first automatically detect all bird vocalizations and then, with the use of 'weakly' labelled recordings, classify them. Evaluations of our proposed method show that it achieves a correct classification of 61% when used in a synthet…
Speech sounds of spoken language are obtained by varying configuration of the articulators surrounding the vocal tract. They contain abundant information that can be utilized to better understand the underlying mechanism of human speech production. We propose a novel deep neural network-based learning framework that understands acoustic information in the variable-length sequence of vocal tract shaping during speech production, captured by real-time magnetic resonance imaging (rtMRI), and translate it into text. The proposed framework comprises of spatiotemporal convolutions, a recurrent netwo…
MyGardenBird: A Machine-Learning-Ready Bird Sound Dataset for Twelve Common Malaysian Birds
Faculty of Computer Science and Information Technology, Universiti Malaya（马来大学计算机科学与信息技术学院）；Faculty of Electrical Engineering, Universiti Teknologi Malaysia（技术学院电气工程学院）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
The Generation and propagation of the human voice is studied in two-dimensions using a full-body domain, using direct numerical simulation. The fluid/air in the vocal tract is modeled as a compressible and viscous fluid interacting with the non-linear, viscoelastic vocal folds (VF). The VF tissue material properties are multi-layered, with varying stiffness, and a finite-strain model is utilized and implemented in a quadratic finite element code. The fluid-solid domains are coupled through a boundary-fitted interface and utilize a Poisson equation-based mesh deformation method. The full-body d…
Uncertainty Calibration of Multi-Label Bird Sound Classifiers
INS, Kiel University, Germany（基尔大学）；Tilburg University, Netherlands（蒂尔堡大学）；Naturalis Biodiversity Center, Netherlands（自然史生物多样性中心）；IES, University of Kassel, Germany（卡塞尔大学）
Queen Mary University of London（伦敦玛丽女王大学）；Charles University（查理大学）；The Czech Academy of Sciences（捷克科学院）；Czech University of Life Sciences Prague（布拉格捷克生命科学大学）；Adam Mickiewicz University（亚当·密茨凯维奇大学）
Many animals emit vocal sounds which, independently from the sounds' function, embed some individually-distinctive signature. Thus the automatic recognition of individuals by sound is a potentially powerful tool for zoology and ecology research and practical monitoring. Here we present a general automatic identification method, that can work across multiple animal species with various levels of complexity in their communication systems. We further introduce new analysis techniques based on dataset manipulations that can evaluate the robustness and generality of a classifier. By using these tec…
