Large language models (LLMs) prompted with text and audio have achieved state-of-the-art performance across various auditory tasks, including speech, music, and general audio, showing emergent abilities on unseen tasks. However, their potential has yet to be fully demonstrated in bioacoustics tasks, such as detecting animal vocalizations in large recordings, classifying rare and endangered species, and labeling context and behavior -- tasks that are crucial for conservation, biodiversity monitoring, and animal behavior studies. In this work, we present NatureLM-audio, the first audio-language foundation model specifically designed for bioacoustics. Our training dataset consists of carefully curated text-audio pairs spanning bioacoustics, speech, and music, designed to address the field's limited availability of annotated data. We demonstrate successful transfer of learned representations from music and speech to bioacoustics, and our model shows promising generalization to unseen taxa and tasks. We evaluate NatureLM-audio on a novel benchmark (BEANS-Zero) and it sets a new state of the art on several bioacoustics tasks, including zero-shot classification of unseen species. To advance bioacoustics research, we release our model weights, benchmark data, and open-source the code for training and benchmark data generation and model training.
Leveraging tropical reef, bird and unrelated sounds for superior transfer learning in marine bioacoustics
University College London（伦敦大学学院）；Google DeepMind（谷歌DeepMind）；Zoological Society of London（伦敦动物学会）；Conservation Metrics Inc.（保护指标公司）；University of New Hampshire（新罕布什尔大学）；K. Lisa Yang Center for Conservation Bioacoustics, Cornell Lab of Ornithology, Cornell University（康奈尔大学鸟类学实验室K. Lisa Yang保护生物声学中心）；Department of Public and Ecosystem Health, Cornell University（康奈尔大学公共与生态系统健康系）；Sound Ocean Science（海洋声学科学）；University of Exeter（埃克塞特大学）；University of Bristol（布里斯托大学）；IPB University（IPB大学）；Google Research（谷歌研究院）
animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacoustics
Department for the Ecology of Animal Societies, Max Planck Institute of Animal Behavior（动物社会生态学部，动物行为马克斯·普朗克研究所）；Department of Biology, University of Konstanz（康斯坦茨大学生物系）；Centre for the Advanced Study of Collective Behaviour, University of Konstanz（集体行为高级研究所以及康斯坦茨大学）；Kalahari Research Centre, Van Zylsrus, Northern Cape, South Africa（卡拉哈里研究中心，南非北开普省范·齐尔鲁斯）；Royal Botanic Gardens Kew, Richmond, UK（皇家植物园肯特，英国里士满）；Zukunftskolleg, University of Konstanz（未来学院，康斯坦茨大学）；Department of Evolutionary Biology and Environmental Studies, University of Zurich（进化生物学与环境研究部，苏黎世大学）；Department of Cognitive Science and Artificial Intelligence, Tilburg University（认知科学与人工智能系，蒂尔堡大学）；Naturalis Biodiversity Center, Leiden, The Netherlands（自然史博物馆生物多样性研究中心，莱顿，荷兰）；Interdisciplinary Center for the Evolution of Language, University of Zurich（语言进化跨学科研究中心，苏黎世大学）；Department of Computer Science, San Diego State University（计算机科学系，圣地亚哥州立大学）
Exploring bat song syllable representations in self-supervised audio encoders
Institute for Logic, Language and Computation, University of Amsterdam（阿姆斯特丹大学逻辑、语言与计算研究所）；Museum für Naturkunde, Leibniz Institute for Evolution and Biodiversity Science（莱布尼茨进化与生物多样性科学自然博物馆）
Lightweight Hopfield Neural Networks for Bioacoustic Detection and Call Monitoring of Captive Primates
School of Engineering, Computing and Mathematical Sciences（工程、计算与数学科学学院）；School of Pharmacy and Life Sciences（药学与生命科学学院）；University of Wolverhampton（沃里克大学）
Parallel Delayed Memory Units for Enhanced Temporal Modeling in Biomedical and Bioacoustic Signal Analysis
Department of Information Technology, WAVES Research Group, Ghent University（信息科技系、WAVES研究组、根特大学）；Department of Electrical and Electronic Engineering, Imperial College London（电子与电气工程系、伦敦帝国学院）
Benchmarking Language Modeling for Lossless Compression of Full-Fidelity Audio
University of California, San Diego, Computer Science and Engineering Department（加州大学圣地亚哥分校计算机科学与工程系）；Carnegie Mellon University, School of Computer Science（卡内基梅隆大学计算机科学学院）
Classifying bioacoustic data without individual call annotations using temporal convolutional networks and feature extractors
Sea Mammal Research Unit, School of Biology, University of St. Andrews（海洋哺乳动物研究单位，生物学系，圣安德鲁大学）；Electrical and Electronic Engineering, University of Strathclyde（电子与电气工程，斯特拉斯克莱德大学）
Western Mediterranean wetlands bird species classification: evaluating small-footprint deep learning approaches on a new annotated dataset
La Salle-Universitat Ramon Llull（拉蒙·柳利大学拉萨尔学院）
部署一个运行在无线声学传感器网络上的专家系统，该网络由生物声学监测设备组成，可以从声音中识别鸟类，这将使许多具有生态价值的任务实现自动化，包括分析鸟类种群组成或检测环境利益领域的濒危物种。得益于人工智能的最新进展，赋予这些设备准确的音频分类功能是可能的，其中深度学习技术表现出色。然而，使生物声学设备负担得起的一个关键问题是使用可以嵌入在资源和电池受限的硬件平台中的小足迹深度神经网络。出于这个原因，这项工作提出了两个重型和大型足迹深度神经网络（VGG 16和ResNet 50）和轻量级替代方案MobileNetV2之间的关键比较分析。我们的实验结果表明，MobileNetV2的平均F1分数比ResNet50低不到5%（0.789 vs. 0.834），表现优于VGG16，占用空间小近40倍。此外，为了比较模型，我们创建并公开了西地中海湿地鸟类数据集，包括201.6分钟和5，795个Aiguamolls de l'Empordà自然公园20种特有鸟类的音频摘录。
Whombat: An open-source annotation tool for machine learning development in bioacoustics
University College London（伦敦大学学院）；University of Edinburgh（爱丁堡大学）；Oregon State University（俄勒冈州立大学）；USDA Forest Service（美国农业部林务局）；Bat Conservation Trust（蝙蝠保护信托基金）
Uncertainty Calibration of Multi-Label Bird Sound Classifiers
INS, Kiel University, Germany（基尔大学）；Tilburg University, Netherlands（蒂尔堡大学）；Naturalis Biodiversity Center, Netherlands（自然史生物多样性中心）；IES, University of Kassel, Germany（卡塞尔大学）
Detection and Classification of Cetacean Echolocation Clicks using Image-based Object Detection Methods applied to Advanced Wavelet-based Transformations
