Many biological monitoring projects rely on acoustic detection of birds. Despite increasingly large datasets, this detection is often manual or semi-automatic, requiring manual tuning/postprocessing. We review the state of the art in automatic bird sound detection, and identify a widespread need for tuning-free and species-agnostic approaches. We introduce new datasets and an IEEE research challenge to address this need, to make possible the development of fully automatic algorithms for bird sound detection.
Machine Listening Lab, Centre for Digital Music (C4DM), Queen Mary Univ. of London, UK Centre d'Ecologie et des Sciences de la Conservation (CESCO), Mus\' e um National d'Histoire Naturelle, CNRS, Sorbonne Univ., Paris, France Centre d'Ecologie Fonctionnelle et Evolutive (CEFE), CNRS, Univ. de Montpellier, Univ. Paul-Val\' e ry Montpellier, France AGH Univ. of Science；Technology, Department of Mechanics；Vibroacoustics, Krak\' o w, Poland Univ. Toulon, Aix Marseille Univ., CNRS, LIS, DYNI team, SABIOD, Marseille, France
Recent advances in birdsong detection and classification have approached a limit due to the lack of fully annotated recordings. In this paper, we present NIPS4Bplus, the first richly annotated birdsong audio dataset, that is comprised of recordings containing bird vocalisations along with their active species tags plus the temporal annotations acquired for them. Statistical information about the recordings, their species specific tags and their temporal annotations are presented along with example uses. NIPS4Bplus could be used in various ecoacoustic tasks, such as training models for bird pop…
Automatic acoustic detection of birds through deep learning: the first Bird Audio Detection challenge
Queen Mary University of London（伦敦玛丽女王大学）；University of Salford（索尔福德大学）；AGH University of Science and Technology（AGH科技大学）；University of Crete（克里特大学）；University of Toulon（土伦大学）
School of Engineering and Applied Sciences, Harvard University（哈佛大学工程与应用科学学院）；K. Lisa Yang Center for Conservation Bioacoustics, Cornell Lab of Ornithology, Cornell University（康奈尔大学鸟类学实验室K. Lisa Yang保护生物声学中心）
Monitoring of bird populations has played a vital role in conservation efforts and in understanding biodiversity loss. The automation of this process has been facilitated by both sensing technologies, such as passive acoustic monitoring, and accompanying analytical tools, such as deep learning. However, machine learning models frequently have difficulty generalizing to examples not encountered in the training data. In our work, we present a semi-supervised approach to identify characteristic calls and environmental noise. We utilize several methods to learn a latent representation of audio sam…
Institut Systématique, Évolution, Biodiversité (ISYEB), Muséum national d’Histoire naturelle, CNRS, Sorbonne Université, EPHE, Université des Antilles（系统学、进化与生物多样性研究所（ISYEB），法国国家自然历史博物馆，法国国家科学研究中心，索邦大学，巴黎高等研究实践学院，安第列斯大学）
Open audio databases such as Xeno-Canto are widely used to build datasets to explore bird song repertoire or to train models for automatic bird sound classification by deep learning algorithms. However, such databases suffer from the fact that bird sounds are weakly labelled: a species name is attributed to each audio recording without timestamps that provide the temporal localization of the bird song of interest. Manual annotations can solve this issue, but they are time consuming, expert-dependent, and cannot run on large datasets. Another solution consists in using a labelling function that…
SEABAD: A Tropical Bird Activity Detection Dataset for Passive Acoustic Monitoring
Faculty of Computer Science and Information Technology, Universiti Malaya（马来亚大学计算机科学与信息技术学院）；Faculty of Electrical Engineering, Universiti Teknologi Malaysia（马来西亚科技大学电气工程学院）
DB3V: A Dialect Dominated Dataset of Bird Vocalisation for Cross-corpus Bird Species Recognition
University of Augsburg（奥格斯堡大学）；Beijing Forestry University（北京林业大学）；Technical University of Munich（慕尼黑工业大学）；Hume AI（休谟人工智能公司）；Imperial College London（伦敦帝国理工学院）；Munich Data Science Institute（慕尼黑数据科学研究所）；Munich Center for Machine Learning（慕尼黑机器学习中心）
Cross-Dialect Bird Species Recognition with Dialect-Calibrated Augmentation
GLAM, Department of Computing, Imperial College London, UK（GLAM，计算系，帝国理工学院伦敦校区，英国）；CHI - Chair of Health Informatics, TUM University Hospital, Munich, Germany（健康信息学主席，技术大学慕尼黑医院，慕尼黑，德国）；MDSI – Munich Data Science Institute, Munich, Germany（慕尼黑数据科学研究所，慕尼黑，德国）；MCML – Munich Center for Machine Learning, Munich, Germany（慕尼黑机器学习中心，慕尼黑，德国）
University of Padua（帕多瓦大学）；Missouri State University（密苏里州立大学）；Tampere University（坦佩雷大学）
In this paper, ensembles of classifiers that exploit several data augmentation techniques and four signal representations for training Convolutional Neural Networks (CNNs) for audio classification are presented and tested on three freely available audio classification datasets: i) bird calls, ii) cat sounds, and iii) the Environmental Sound Classification dataset. The best performing ensembles combining data augmentation techniques with different signal representations are compared and shown to outperform the best methods reported in the literature on these datasets. The approach proposed here…
Audio denoising has been explored for decades using both traditional and deep learning-based methods. However, these methods are still limited to either manually added artificial noise or lower denoised audio quality. To overcome these challenges, we collect a large-scale natural noise bird sound dataset. We are the first to transfer the audio denoising problem into an image segmentation problem and propose a deep visual audio denoising (DVAD) model. With a total of 14,120 audio images, we develop an audio ImageMask tool and propose to use a few-shot generalization strategy to label these imag…
A strongly annotated passive acoustic dataset for tropical bird monitoring
Microsoft AI for Good Research Lab（微软AI for Good研究实验室）；Instituto de Investigación de Recursos Biológicos Alexander von Humboldt（亚历山大冯·洪堡生物资源研究所）；Center for Research and Formation in Artificial Intelligence（人工智能研究与培养中心）；Fundación Manacus（曼卡斯基金会）；Louisiana State University（路易斯安那州立大学）；Museum of Natural Sciences（自然博物馆）
This paper studies the detection of bird calls in audio segments using stacked convolutional and recurrent neural networks. Data augmentation by blocks mixing and domain adaptation using a novel method of test mixing are proposed and evaluated in regard to making the method robust to unseen data. The contributions of two kinds of acoustic features (dominant frequency and log mel-band energy) and their combinations are studied in the context of bird audio detection. Our best achieved AUC measure on five cross-validations of the development data is 95.5% and 88.1% on the unseen evaluation data.
Bird sounds possess distinctive spectral structure which may exhibit small shifts in spectrum depending on the bird species and environmental conditions. In this paper, we propose using convolutional recurrent neural networks on the task of automated bird audio detection in real-life environments. In the proposed method, convolutional layers extract high dimensional, local frequency shift invariant features, while recurrent layers capture longer term dependencies between the features extracted from short time frames. This method achieves 88.5% Area Under ROC Curve (AUC) score on the unseen eva…
Audio sound recognition and classification is used for many tasks and applications including human voice recognition, music recognition and audio tagging. In this paper we apply Mel Frequency Cepstral Coefficients (MFCC) in combination with a range of machine learning models to identify (Australian) birds from publicly available audio files of their birdsong. We present approaches used for data processing and augmentation and compare the results of various state of the art machine learning models. We achieve an overall accuracy of 91% for the top-5 birds from the 30 selected as the case study.…
Technological Educational Institute of Crete（克里特技术教育学院）
This work focuses on reliable detection and segmentation of bird vocalizations as recorded in the open field. Acoustic detection of avian sounds can be used for the automatized monitoring of multiple bird taxa and querying in long-term recordings for species of interest. These tasks are tackled in this work, by suggesting two approaches: A) First, DenseNets are applied to weekly labeled data to infer the attention map of the dataset (i.e. Salience and CAM). We push further this idea by directing attention maps to the YOLO v2 Deepnet-based, detection framework to localize bird vocalizations. B)…
University of Würzburg（维尔茨堡大学）；Institute of Computer Science, University of Würzburg（维尔茨堡大学计算机科学研究所）
It is easier to hear birds than see them. However, they still play an essential role in nature and are excellent indicators of deteriorating environmental quality and pollution. Recent advances in Deep Neural Networks allow us to process audio data to detect and classify birds. This technology can assist researchers in monitoring bird populations and biodiversity. We propose a sound detection and classification pipeline to analyze complex soundscape recordings and identify birdcalls in the background. Our method learns from weak labels and few data and acoustically recognizes the bird species.…
Detecting bird sounds in audio recordings automatically, if accurate enough, is expected to be of great help to the research community working in bio- and ecoacoustics, interested in monitoring biodiversity based on audio field recordings. To estimate how accurate the state-of-the-art machine learning approaches are, the Bird Audio Detection challenge involving large audio datasets was recently organized. In this paper, experiments using several types of convolutional neural networks (i.e. standard CNNs, residual nets and densely connected nets) are reported in the framework of this challenge.…
Citizen Science aims to engage people in research activities on important issues related to their well-being. Smart Cities aim to provide them with services that improve the quality of their life. Both concepts have seen significant growth in the last years, and can be further enhanced by combining their purposes with IoT technologies that allow for dynamic and large-scale communication and interaction. However, exciting and retaining the interest of participants is a key factor for such initiatives. In this paper we suggest that engagement in Citizen Science projects applied on Smart Cities i…
Digital audio processing tools offer music researchers the opportunity to examine both non-notated music and music as performance. This chapter summarises the types of information that can be extracted from audio as well as currently available audio tools for music corpus studies. The survey of extraction methods includes both a primer on signal processing and background theory on audio feature extraction. The survey of audio tools focuses on widely used tools, including both those with a graphical user interface, namely Audacity and Sonic Visualiser, and code-based tools written in the C/C++,…
Technological Educational Institute of Crete（克里特技术教育学院）
This work focuses on reliable detection of bird sound emissions as recorded in the open field. Acoustic detection of avian sounds can be used for the automatized monitoring of multiple bird taxa and querying in long-term recordings for species of interest for researchers, conservation practitioners, and decision makers. Recordings in the wild can be very noisy due to the exposure of the microphones to a large number of audio sources originating from all distances and directions, the number and identity of which cannot be known a-priori. The co-existence of the target vocalizations with abiotic…
University of Pretoria（比勒陀利亚大学）；Dedan Kimathi University of Technology（德丹·基马蒂科技大学）
This paper is an investigation into aspects of an audio classification pipeline that will be appropriate for the monitoring of bird species on edges devices. These aspects include transfer learning, data augmentation and model optimization. The hope is that the resulting models will be good candidates to deploy on edge devices to monitor bird populations. Two classification approaches will be taken into consideration, one which explores the effectiveness of a traditional Deep Neural Network(DNN) and another that makes use of Convolutional layers.This study aims to contribute empirical evidence…
Cornell University（康奈尔大学）；Google（谷歌）；University of Edinburgh（爱丁堡大学）；University of Copenhagen（哥本哈根大学）
We present a new benchmark dataset, Sapsucker Woods 60 (SSW60), for advancing research on audiovisual fine-grained categorization. While our community has made great strides in fine-grained visual categorization on images, the counterparts in audio and video fine-grained categorization are relatively unexplored. To encourage advancements in this space, we have carefully constructed the SSW60 dataset to enable researchers to experiment with classifying the same set of categories in three different modalities: images, audio, and video. The dataset covers 60 species of birds and is comprised of i…
Ultra-Compact CNN Architectures for Tropical Bird Audio Detection on Microcontrollers
Faculty of Computer Science and Information Technology, Universiti Malaya（马来西亚大学计算机科学与信息技术学院）；Faculty of Electrical Engineering, Universiti Teknologi Malaysia（马来西亚理工大学电气工程学院）
IIT Mandi（曼迪印度理工学院）；University of Oxford（牛津大学）；Idiap Research Institute（伊迪亚普研究所）
In this work, we propose a supervised, convex representation based audio hashing framework for bird species classification. The proposed framework utilizes archetypal analysis, a matrix factorization technique, to obtain convex-sparse representations of a bird vocalization. These convex representations are hashed using Bloom filters with non-cryptographic hash functions to obtain compact binary codes, designated as conv-codes. The conv-codes extracted from the training examples are clustered using class-specific k-medoids clustering with Jaccard coefficient as the similarity metric. A hash tab…
Based on the transfer learning, we design a bird species identification model that uses the VGG-16 model (pretrained on ImageNet) for feature extraction, then a classifier consisting of two fully-connected hidden layers and a Softmax layer is attached. We compare the performance of the proposed model with the original VGG16 model. The results show that the former has higher train efficiency, but lower mean average precisions(MAP). To improve the MAP of the proposed model, we investigate the result fusion mode to form multi-channel identification model, the best MAP reaches 0.9998. The number o…
Vienna University of Technology（维也纳工业大学）；Austrian Institute of Technology（奥地利技术研究所）
We present a multi-modal Deep Neural Network (DNN) approach for bird song identification. The presented approach takes both audio samples and metadata as input. The audio is fed into a Convolutional Neural Network (CNN) using four convolutional layers. The additionally provided metadata is processed using fully connected layers. The flattened convolutional layers and the fully connected layer of the metadata are joined and fed into a fully connected layer. The resulting architecture achieved 2., 3. and 4. rank in the BirdCLEF2017 task in various training configurations.
Concordia University（康科迪亚大学）；Mila-Quebec AI Institute（魁北克人工智能研究所）；The Hebrew University of Jerusalem（特拉维夫大学）；University of Cambridge（剑桥大学）；Indiana University（印第安纳大学）；Carnegie Mellon University（卡内基梅隆大学）；Microsoft（微软公司）；Université de Montréal（蒙特利尔大学）；Université de Toulon（托尔努瓦大学）；Google（谷歌公司）；Apple（苹果公司）；Laval University（拉瓦尔大学）；National Taiwan University（台湾大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Bangladesh University of Engineering and Technology（孟加拉国工程技术大学）
Identification of bird species from audio records is one of the challenging tasks due to the existence of multiple species in the same recording, noise in the background, and long-term recording. Besides, choosing a proper acoustic feature from audio recording for bird species classification is another problem. In this paper, a hybrid method is represented comprising both traditional signal processing and a deep learning-based approach to classify bird species from audio recordings of diverse sources and types. Besides, a detailed study with 34 different features helps to select the proper fea…
Recording and analysing environmental audio recordings has become a common approach for monitoring the environment. A current problem with performing analyses of environmental recordings is interference from noise that can mask sounds of interest. This makes detecting these sounds more difficult and can require additional resources. While some work has been done to remove stationary noise from environmental recordings, there has been little effort to remove noise from non-stationary sources, such as rain, wind, engines, and animal vocalisations that are not of interest. In this paper, we addre…
Killing two birds with one stone: Can an audio captioning system also be used for audio-text retrieval?
IRIT（图卢兹计算机科学研究所）；Université Paul Sabatier（保罗·萨巴蒂埃大学）；CNRS（法国国家科学研究中心）；Artificial and Natural Intelligence Toulouse Institute (ANITI)（图卢兹人工智能与自然智能研究所（ANITI））
A Bird Song Detector for improving bird identification through Deep Learning: a case study from Doñana
University of Córdoba（科尔多瓦大学）；Estación Biológica de Doñana（多纳纳生物站）；University of Cádiz（卡迪斯大学）；Universidad Politécnica de Madrid（马德里理工大学）；University of Sevilla（塞维利亚大学）
The deployment of an expert system running over a wireless acoustic sensors network made up of bioacoustic monitoring devices that recognise bird species from their sounds would enable the automation of many tasks of ecological value, including the analysis of bird population composition or the detection of endangered species in areas of environmental interest. Endowing these devices with accurate audio classification capabilities is possible thanks to the latest advances in artificial intelligence, among which deep learning techniques excel. However, a key issue to make bioacoustic devices af…
Leveraging tropical reef, bird and unrelated sounds for superior transfer learning in marine bioacoustics
University College London（伦敦大学学院）；Google DeepMind（谷歌DeepMind）；Zoological Society of London（伦敦动物学会）；Conservation Metrics Inc.（保护指标公司）；University of New Hampshire（新罕布什尔大学）；K. Lisa Yang Center for Conservation Bioacoustics, Cornell Lab of Ornithology, Cornell University（康奈尔大学鸟类学实验室K. Lisa Yang保护生物声学中心）；Department of Public and Ecosystem Health, Cornell University（康奈尔大学公共与生态系统健康系）；Sound Ocean Science（海洋声学科学）；University of Exeter（埃克塞特大学）；University of Bristol（布里斯托大学）；IPB University（IPB大学）；Google Research（谷歌研究院）
We introduce a novel approach to studying animal behaviour and the context in which it occurs, through the use of microphone backpacks carried on the backs of individual free-flying birds. These sensors are increasingly used by animal behaviour researchers to study individual vocalisations of freely behaving animals, even in the field. However such devices may record more than an animals vocal behaviour, and have the potential to be used for investigating specific activities (movement) and context (background) within which vocalisations occur. To facilitate this approach, we investigate the au…
Event detection improves when events are captured by two different modalities rather than just one. But to train detection systems on multiple modalities is challenging, in particular when there is abundance of unlabelled data but limited amounts of labeled data. We develop a novel self-supervised learning technique for multi-modal data that learns (hidden) correlations between simultaneously recorded microphone (sound) signals and accelerometer (body vibration) signals. The key objective of this work is to learn useful embeddings associated with high performance in downstream event detection …
University of Electronic Science and Technology of China（电子科技大学）
With the development of deep learning and artificial intelligence, audio synthesis has a pivotal role in the area of machine learning and shows strong applicability in the industry. Meanwhile, significant efforts have been dedicated by researchers to handle multimodal tasks at present such as audio-visual multimodal processing. In this paper, we conduct a survey on audio synthesis and audio-visual multimodal processing, which helps understand current research and future trends. This review focuses on text to speech(TTS), music generation and some tasks that combine visual and acoustic informat…
MyGardenBird: A Machine-Learning-Ready Bird Sound Dataset for Twelve Common Malaysian Birds
Faculty of Computer Science and Information Technology, Universiti Malaya（马来大学计算机科学与信息技术学院）；Faculty of Electrical Engineering, Universiti Teknologi Malaysia（技术学院电气工程学院）
Uncertainty Calibration of Multi-Label Bird Sound Classifiers
INS, Kiel University, Germany（基尔大学）；Tilburg University, Netherlands（蒂尔堡大学）；Naturalis Biodiversity Center, Netherlands（自然史生物多样性中心）；IES, University of Kassel, Germany（卡塞尔大学）
