Synthesizing and converting environmental sounds have the potential for many applications such as supporting movie and game production, data augmentation for sound event detection and scene classification. Conventional works on synthesizing and converting environmental sounds are based on a physical modeling or concatenative approach. However, there are a limited number of works that have addressed environmental sound synthesis and conversion with statistical generative models; thus, this research area is not yet well organized. In this paper, we review problem definitions, applications, and evaluation methods of environmental sound synthesis and conversion. We then report on environmental sound synthesis using sound event labels, in which we focus on the current performance of statistical environmental sound synthesis and investigate how we should conduct subjective experiments on environmental sound synthesis.
Ritsumeikan University（立命馆大学）；Doshisha University（同志社大学）；The University of Tokyo（东京大学）
Although several methods of environmental sound synthesis have been proposed, there has been no discussion on how synthesized environmental sounds should be evaluated. Only either subjective or objective evaluations have been conducted in conventional evaluations, and it is not clear what type of evaluation should be carried out. In this paper, we investigate how to evaluate synthesized environmental sounds. We also propose a subjective evaluation methodology to evaluate whether the synthesized sound appropriately represents the information input to the environmental sound synthesis system. In…
Spatial Analysis and Synthesis Methods: Subjective and Objective Evaluations Using Various Microphone Arrays in the Auralization of a Critical Listening Room
University of Huddersfield（哈德斯菲尔德大学）；Genelec OY（吉力科技公司）
Carnegie Mellon University（卡内基梅隆大学）；Microsoft（微软）；Mohammed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）
Machine Listening, as usually formalized, attempts to perform a task that is, from our perspective, fundamentally human-performable, and performed by humans. Current automated models of Machine Listening vary from purely data-driven approaches to approaches imitating human systems. In recent years, the most promising approaches have been hybrid in that they have used data-driven approaches informed by models of the perceptual, cognitive, and semantic processes of the human system. Not only does the guidance provided by models of human perception and domain knowledge enable better, and more gen…
Environmental Sound Deepfake Detection Challenge: An Overview
1 School of Electrical Engineering, KAIST, Daejeon, Republic of Korea 2 University of Melbourne, Australia, 3 Fortemedia Singapore, Singapore, 4 Xi’an University of Posts \& Telecommunications, Xi’an, China, 5 Xi'an Lianfeng Acoustic Technologies Co., Ltd., China
Subjective evaluations are critical for assessing the perceptual realism of sounds in audio-synthesis driven technologies like augmented and virtual reality. However, they are challenging to set up, fatiguing for users, and expensive. In this work, we tackle the problem of capturing the perceptual characteristics of localizing sounds. Specifically, we propose a framework for building a general purpose quality metric to assess spatial localization differences between two binaural recordings. We model localization similarity by utilizing activation-level distances from deep networks trained for …
Ritsumeikan University（立命馆大学）；Doshisha University（同志社大学）；The University of Tokyo（东京大学）；Kansai University（关西大学）
In this paper, we propose a framework for environmental sound synthesis from onomatopoeic words. As one way of expressing an environmental sound, we can use an onomatopoeic word, which is a character sequence for phonetically imitating a sound. An onomatopoeic word is effective for describing diverse sound features. Therefore, using onomatopoeic words for environmental sound synthesis will enable us to generate diverse environmental sounds. To generate diverse sounds, we propose a method based on a sequence-to-sequence framework for synthesizing environmental sounds from onomatopoeic words. We…
This paper introduces the Procedural (audio) Variational autoEncoder (ProVE) framework as a general approach to learning Procedural Audio PA models of environmental sounds with an improvement to the realism of the synthesis while maintaining provision of control over the generated sound through adjustable parameters. The framework comprises two stages: (i) Audio Class Representation, in which a latent representation space is defined by training an audio autoencoder, and (ii) Control Mapping, in which a joint function of static/temporal control variables derived from the audio and a random samp…
Overview of ESDD2: Environment-Aware Speech and Sound Deepfake Detection Challenge
Duke Kunshan University（昆山杜克大学）；Korea Advanced Institute of Science and Technology（韩国科学技术院）；The University of Melbourne（墨尔本大学）；Johns Hopkins University（约翰霍普金斯大学）；Fortemedia Singapore（Fortemedia新加坡）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
A deep neural network solution for time-scale modification (TSM) focused on large stretching factors is proposed, targeting environmental sounds. Traditional TSM artifacts such as transient smearing, loss of presence, and phasiness are heavily accentuated and cause poor audio quality when the TSM factor is four or larger. The weakness of established TSM methods, often based on a phase vocoder structure, lies in the poor description and scaling of the transient and noise components, or nuances, of a sound. Our novel solution combines a sines-transients-noise decomposition with an independent Wa…
Graduate School of Media and Governance, Keio University（庆应义塾大学媒体与治理研究科）；Faculty of Environment and Information Studies, Keio University（庆应义塾大学环境信息学部）；NTT Communication Science Laboratories（NTT通信科学研究所）；School of Psychology, Chukyo University（中京大学心理学部）
Auditory frisson is the experience of feeling of cold or shivering related to sound in the absence of a physical cold stimulus. Multiple examples of frisson-inducing sounds have been reported, but the mechanism of auditory frisson remains elusive. Typical frisson-inducing sounds may contain a looming effect, in which a sound appears to approach the listener's peripersonal space. Previous studies on sound in peripersonal space have provided objective measurements of sound-inducing effects, but few have investigated the subjective experience of frisson-inducing sounds. Here we explored whether i…
Carl von Ossietzky University Oldenburg（奥尔登堡大学）；Fraunhofer IDMT（弗劳恩霍夫数字媒体技术研究所）
Listening to the audio of TV broadcast signals can be challenging for hearing-impaired as well as normal-hearing listeners, especially when background sounds are prominent or too loud compared to the speech signal. This can result in a reduced satisfaction and increased listening effort of the listeners. Since the broadcast sound is usually premixed, we perform a subjective evaluation for quantifying the potential of speech enhancement systems based on audio source separation and recurrent neural networks (RNN). Recently, RNNs have shown promising results in the context of sound source separat…
Audio quality assessment is critical for assessing the perceptual realism of sounds. However, the time and expense of obtaining ''gold standard'' human judgments limit the availability of such data. For AR&VR, good perceived sound quality and localizability of sources are among the key elements to ensure complete immersion of the user. Our work introduces SAQAM which uses a multi-task learning framework to assess listening quality (LQ) and spatialization quality (SQ) between any given pair of binaural signals without using any subjective data. We model LQ by training on a simulated dataset of …
Sound event localization and detection is a novel area of research that emerged from the combined interest of analyzing the acoustic scene in terms of the spatial and temporal activity of sounds of interest. This paper presents an overview of the first international evaluation on sound event localization and detection, organized as a task of the DCASE 2019 Challenge. A large-scale realistic dataset of spatialized sound events was generated for the challenge, to be used for training of learning-based approaches, and for evaluation of the submissions in an unlabeled subset. The overview presents…
Formula-Supervised Sound Event Detection: Pre-Training Without Real Data
Keio University（庆应义塾大学）；National Institute of Advanced Industrial Science and Technology (AIST)（独立行政法人产业技术综合研究所）；Waseda University（早稻田大学）；Doshisha University（同志社大学）；University of Oxford（牛津大学）
Indian Institute of Science（印度科学学院）；Voice and Speech Systems（语音与言语系统公司）
Subjective and objective experiments are conducted to understand the extent to which a speaker's gender influences the acoustics of unvoiced (U) sounds. U segments of utterances are replaced by the corresponding segments of a speaker of opposite gender to prepare modified utterances. Humans are asked to judge if the modified utterance is spoken by one or two speakers. The experiments show that human subjects are unable to distinguish the modified from the original. Thus, listeners are able to identify the U segments irrespective of the gender, which may be based on some speaker-independent inv…
Environmental Sound Classification (ESC) is an active research area in the audio domain and has seen a lot of progress in the past years. However, many of the existing approaches achieve high accuracy by relying on domain-specific features and architectures, making it harder to benefit from advances in other fields (e.g., the image domain). Additionally, some of the past successes have been attributed to a discrepancy of how results are evaluated (i.e., on unofficial splits of the UrbanSound8K (US8K) dataset), distorting the overall progression of the field. The contribution of this paper is t…
Drum-to-Vocal Percussion Sound Conversion and Its Evaluation Methodology
Graduate School of Information Science and Technology, University of Tokyo（东京大学信息科学与技术研究生院）；National Institute of Advanced Industrial Science and Technology (AIST)（国家先进工业科学与技术研究院）；Faculty of Science and Technology, Keio University（庆应大学理工学院）
The University of Tokyo（东京大学）；Imperial College London（帝国理工学院）
Computer audition (CA) has been demonstrated to be efficient in healthcare domains for speech-affecting disorders (e.g., autism spectrum, depression, or Parkinson's disease) and body sound-affecting abnormalities (e. g., abnormal bowel sounds, heart murmurs, or snore sounds). Nevertheless, CA has been underestimated in the considered data-driven technologies for fighting the COVID-19 pandemic caused by the SARS-CoV-2 coronavirus. In this light, summarise the most recent advances in CA for COVID-19 speech and/or sound analysis. While the milestones achieved are encouraging, there are yet not an…
Imperial College London（伦敦帝国学院）；University of Augsburg（奥格斯堡大学）；audEERING GmbH（audEERING有限公司）；The University of Tokyo（东京大学）
At the time of writing, the world population is suffering from more than 10,000 registered COVID-19 disease epidemic induced deaths since the outbreak of the Corona virus more than three months ago now officially known as SARS-CoV-2. Since, tremendous efforts have been made worldwide to counter-steer and control the epidemic by now labelled as pandemic. In this contribution, we provide an overview on the potential for computer audition (CA), i.e., the usage of speech and sound analysis by artificial intelligence to help in this scenario. We first survey which types of related or contextually s…
National Institute of Technology, Tokuyama College（国立技术院德山高专）；NTT Media Intelligence Laboratories（NTT媒体智能实验室）；Waseda University（早稻田大学）
Improving subjective sound quality of enhanced signals is one of the most important missions in speech enhancement. For evaluating the subjective quality, several methods related to perceptually-motivated objective sound quality assessment (OSQA) have been proposed such as PESQ (perceptual evaluation of speech quality). However, direct use of such measures for training deep neural network (DNN) is not allowed in most cases because popular OSQAs are non-differentiable with respect to DNN parameters. Therefore, the previous study has proposed to approximate the score of OSQAs by an auxiliary DNN…
CNRS（法国国家科学研究中心）；Ecole Centrale Nantes（南特中央理工学院）；Nantes Université（南特大学）；Gaudio Lab, Inc.（Gaudio实验室公司）；KAIST（韩国科学技术院）；Carnegie Mellon University（卡内基梅隆大学）；New York University（纽约大学）；Doshisha University（同志社大学）；The University of Tokyo（东京大学）
Vacuum cleaners are one of the most widely used household appliances associated with unpleasant noises. Previous studies have indicated the severity of vacuum cleaner noise and its impact on the users nearby. The quantified measurements of the generated noise standalone are not sufficient for the selection or designing of vacuum cleaners. The human perception must also be included for a better assessment of the quality of sound. A hybrid approach known as psychoacoustics, which comprises subjective and objective evaluations of sounds, is widely used in recent times. This paper focuses on the e…
Joint Prediction of Audio Event and Annoyance Rating in an Urban Soundscape by Hierarchical Graph Representation Learning
Ghent University（根特大学）；University of Leicester（莱斯特大学）；Shenzhen University（深圳大学）；University College London（伦敦大学学院）；Ghent University-Imec（根特大学-imec）；University of Surrey（萨里大学）
Humans can imagine a scene from a sound. We want machines to do so by using conditional generative adversarial networks (GANs). By applying the techniques including spectral norm, projection discriminator and auxiliary classifier, compared with naive conditional GAN, the model can generate images with better quality in terms of both subjective and objective evaluations. Almost three-fourth of people agree that our model have the ability to generate images related to sounds. By inputting different volumes of the same sound, our model output different scales of changes based on the volumes, show…
In a conventional voice conversion (VC) framework, a VC model is often trained with a clean dataset consisting of speech data carefully recorded and selected by minimizing background interference. However, collecting such a high-quality dataset is expensive and time-consuming. Leveraging crowd-sourced speech data in training is more economical. Moreover, for some real-world VC scenarios such as VC in video and VC-based data augmentation for speech recognition systems, the background sounds themselves are also informative and need to be maintained. In this paper, to explore VC with the flexibil…
Gaudio Lab, Inc.（Gaudio实验室公司）；KAIST（韩国科学技术院）；Carnegie Mellon University（卡内基梅隆大学）；New York University（纽约大学）；Doshisha University（同志社大学）；Ritsumeikan University（立命馆大学）；CNRS（法国国家科学研究中心）；Ecole Centrale Nantes（南特中央理工学院）；The University of Tokyo（东京大学）
Universitat Pompeu Fabra（庞培法布拉大学）；Joint Research Centre, European Commission（欧盟委员会联合研究中心）
We present a deep neural network-based methodology for synthesising percussive sounds with control over high-level timbral characteristics of the sounds. This approach allows for intuitive control of a synthesizer, enabling the user to shape sounds without extensive knowledge of signal processing. We use a feedforward convolutional neural network-based architecture, which is able to map input parameters to the corresponding waveform. We propose two datasets to evaluate our approach on both a restrictive context, and in one covering a broader spectrum of sounds. The timbral features used as par…
The rise of deep learning algorithms has led many researchers to withdraw from using classic signal processing methods for sound generation. Deep learning models have achieved expressive voice synthesis, realistic sound textures, and musical notes from virtual instruments. However, the most suitable deep learning architecture is still under investigation. The choice of architecture is tightly coupled to the audio representations. A sound's original waveform can be too dense and rich for deep learning models to deal with efficiently - and complexity increases training time and computational cos…
Immersive audio-visual perception relies on the spatial integration of both auditory and visual information which are heterogeneous sensing modalities with different fields of reception and spatial resolution. This study investigates the perceived coherence of audiovisual object events presented either centrally or peripherally with horizontally aligned/misaligned sound. Various object events were selected to represent three acoustic feature classes. Subjective test results in a simulated virtual environment from 18 participants indicate a wider capture region in the periphery, with an outward…
Tianjin University（天津大学）；University of Maryland at College Park（马里兰大学帕克分校）
Sound, as a crucial sensory channel, plays a vital role in improving the reality and immersiveness of a virtual environment, following only vision in importance. Sound can provide important clues such as sound directionality and spatial size. This paper gives a broad overview of research works on sound simulation in virtual reality, games, multimedia, computer-aided design. We first survey various sound synthesis methods, including harmonic synthesis, texture synthesis, spectral analysis, and physics-based synthesis. Then, we summarize popular sound propagation techniques, namely wave-based me…
Clustering and Data Augmentation to Improve Accuracy of Sleep Assessment and Sleep Individuality Analysis
Graduate School of Information Science and Technology, Osaka University（大阪大学信息科学与技术研究生院）；SANKEN, Osaka University（大阪大学先进科学研究中心）；Kyoto Tachibana University（京都橘大学）
A Comprehensive Survey on Heart Sound Analysis in the Deep Learning Era
University of Bremen（不来梅大学）；Imperial College London（伦敦帝国学院）；Griffith University（格里菲斯大学）；Beijing Institute of Technology（北京理工大学）；University of Augsburg（奥格斯堡大学）；Technical University of Munich（慕尼黑工业大学）；GLAM, Imperial College London（伦敦帝国学院GLAM机构）
National University of Ireland Galway（爱尔兰国立大学戈尔韦分校）；University Politehnica of Bucharest（布加勒斯特理工大学）
Speech synthesis has come a long way as current text-to-speech (TTS) models can now generate natural human-sounding speech. However, most of the TTS research focuses on using adult speech data and there has been very limited work done on child speech synthesis. This study developed and validated a training pipeline for fine-tuning state-of-the-art (SOTA) neural TTS models using child speech datasets. This approach adopts a multi-speaker TTS retuning workflow to provide a transfer-learning pipeline. A publicly available child speech dataset was cleaned to provide a smaller subset of approximate…
