Automatically identifying, counting, and describing wild animals in camera-trap images with deep learning
University of Wyoming（怀俄明大学）；Auburn University（奥本大学）；Harvard University（哈佛大学）；University of Oxford（牛津大学）；University of Minnesota（明尼苏达大学）；Uber AI Labs（优步人工智能实验室）
Mohammed Sadegh Norouzzadeh, Anh Nguyen, Margaret Kosmala, Ali Swanson, Meredith Palmer, Craig Packer, Jeff Clune
Having accurate, detailed, and up-to-date information about the location and behavior of animals in the wild would revolutionize our ability to study and conserve ecosystems. We investigate the ability to automatically, accurately, and inexpensively collect such data, which could transform many fields of biology, ecology, and zoology into "big data" sciences. Motion sensor "camera traps" enable collecting wildlife pictures inexpensively, unobtrusively, and frequently. However, extracting information from these pictures remains an expensive, time-consuming, manual task. We demonstrate that such information can be automatically extracted by deep learning, a cutting-edge type of artificial intelligence. We train deep convolutional neural networks to identify, count, and describe the behaviors of 48 species in the 3.2-million-image Snapshot Serengeti dataset. Our deep neural networks automatically identify animals with over 93.8% accuracy, and we expect that number to improve rapidly in years to come. More importantly, if our system classifies only images it is confident about, our system can automate animal identification for 99.3% of the data while still performing at the same 96.6% accuracy as that of crowdsourced teams of human volunteers, saving more than 8.4 years (at 40 hours per week) of human labeling effort (i.e. over 17,000 hours) on this 3.2-million-image dataset. Those efficiency gains immediately highlight the importance of using deep neural networks to automate data extraction from camera-trap images. Our results suggest that this technology could enable the inexpensive, unobtrusive, high-volume, and even real-time collection of a wealth of information about vast numbers of animals in the wild.
Politecnico di Milano（米兰理工大学）；University of California, Davis（加州大学戴维斯分校）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
Activity recognition and, more generally, behavior inference tasks are gaining a lot of interest. Much of it is work in the context of human behavior. New available tracking technologies for wild animals are generating datasets that indirectly may provide information about animal behavior. In this work, we propose a method for classifying these data into behavioral annotation, particularly collective behavior of a social group. Our method is based on sequence analysis with a direct encoding of the interactions of a group of wild animals. We evaluate our approach on a real world dataset, showin…
Universidad de Antioquia UdeA（安蒂奥基亚大学（UdeA））；Instituto Tecnológico Metropolitano ITM（大都会理工学院（ITM））
Non intrusive monitoring of animals in the wild is possible using camera trapping framework, which uses cameras triggered by sensors to take a burst of images of animals in their habitat. However camera trapping framework produces a high volume of data (in the order on thousands or millions of images), which must be analyzed by a human expert. In this work, a method for animal species identification in the wild using very deep convolutional neural networks is presented. Multiple versions of the Snapshot Serengeti dataset were used in order to probe the ability of the method to cope with differ…
The University of Sydney（悉尼大学）；Xidian University（西安电子科技大学）；JD Explore Academy（京东探索研究院）
Accurate animal pose estimation is an essential step towards understanding animal behavior, and can potentially benefit many downstream applications, such as wildlife conservation. Previous works only focus on specific animals while ignoring the diversity of animal species, limiting the generalization ability. In this paper, we propose AP-10K, the first large-scale benchmark for mammal animal pose estimation, to facilitate the research in animal pose estimation. AP-10K consists of 10,015 images collected and filtered from 23 animal families and 54 species following the taxonomic rank and high-…
Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Nanjing Institute of Software Technology（南京软件技术研究院）；University of North Texas（北德克萨斯大学）
Multi-animal tracking (MAT), a multi-object tracking (MOT) problem, is crucial for animal motion and behavior analysis and has many crucial applications such as biology, ecology and animal conservation. Despite its importance, MAT is largely under-explored compared to other MOT problems such as multi-human tracking due to the scarcity of dedicated benchmarks. To address this problem, we introduce AnimalTrack, a dedicated benchmark for multi-animal tracking in the wild. Specifically, AnimalTrack consists of 58 sequences from a diverse selection of 10 common animal categories. On average, each s…
SAM 3D Animal: Promptable Animal 3D Reconstruction from Images in the Wild
University of Cambridge（剑桥大学）；Southern University of Science and Technology（南方科技大学）；Tsinghua University（清华大学）；IMATI-CNR, Milan, Italy（意大利米兰IMATI-CNR研究所）
本文提出SAM 3D Animal，首个可提示的多动物3D重建框架，基于SMAL+参数化动物模型，通过关键点和掩码提示实现拥挤和遮挡场景的可靠重建，利用Herd3D数据集在多个数据集上取得最优结果。
Ove Arup and Partners（奥雅纳工程顾问公司）；Newcastle University（纽卡斯尔大学）
Understanding the abundance of a species is the first step towards understanding both its long-term sustainability and the impact that we may be having upon it. Ecologists use camera traps to remotely survey for the presence of specific animal species. Previous studies have shown that deep learning models can be trained to automatically detect and classify animals within camera trap imagery with high levels of confidence. However, the ability to train these models is reliant upon having enough high-quality training data. What happens when the animal is rare or the data sets are non-existent? T…
BuckTales : A multi-UAV dataset for multi-object tracking and re-identification of wild antelopes
Max Planck Institute of Animal Behavior（马克斯·普朗克动物行为研究所）；University of Konstanz（康斯坦茨大学）；Indian Institute of Science（印度科学学院）；BITS Pilani Hyderabad（BITS Pilani海得拉巴校区）
Automatic species classification in camera traps would greatly help the biodiversity monitoring and species analysis in the earth. In order to accelerate the development of automatic species classification task, "Microsoft AI for Earth" have prepared a challenge in FGVC6 workshop at CVPR 2019, which called "iWildCam 2019 competition". In this work, we propose an efficient method for categorizing animals in the wild. We transfer the state-of-the-art ImagaNet pretrained models to the problem. To improve the generalization and robustness of the model, we utilize efficient image augmentation and r…
Multimodal Foundation Models for Zero-shot Animal Species Recognition in Camera Trap Images
Microsoft AI for Good Lab, Redmond（微软人工智能公益实验室）；University of Southern California, Los Angeles（南加州大学）；Microsoft Research, Redmond（微软研究院）；Nanyang Technological University, Singapore（南洋理工大学）；Universidad de Los Andes, Colombia（洛斯安第斯大学）
Unsustainable trade in wildlife is one of the major threats affecting the global biodiversity crisis. An important part of the trade now occurs on the internet, especially on digital marketplaces and social media. Automated methods to identify trade posts are needed as resources for conservation are limited. Here, we developed machine vision models based on Deep Neural Networks with the aim to automatically identify images of exotic pet animals for sale. A new training dataset representing exotic pet animals advertised for sale on the web was generated for this purpose. We trained 24 neural-ne…
University of Udine（乌迪内大学）；University of Edinburgh（爱丁堡大学）；Wildlife Conservation Society（野生生物保护学会）
Photographs of wild animals in their natural habitats can be recorded unobtrusively via cameras that are triggered by motion nearby. The installation of such camera traps is becoming increasingly common across the world. Although this is a convenient source of invaluable data for biologists, ecologists and conservationists, the arduous task of poring through potentially millions of pictures each season introduces prohibitive costs and frustrating delays. We develop automatic algorithms that are able to detect animals, identify the species of animals and to recognize individual animals for two …
Pytorch-Wildlife: A Collaborative Deep Learning Framework for Conservation
Microsoft AI for Good（微软AI公益部门）；Center for Research and Formation in Artificial Intelligence, Universidad de los Andes（安第斯大学人工智能研究与形成中心）；Massachusetts Institute of Technology（麻省理工学院）
TotalEnergies SE（道达尔能源公司）；University of Lille（里尔大学）；CNRS（法国国家科学研究中心）；MODIS
Camera traps have revolutionized the animal research of many species that were previously nearly impossible to observe due to their habitat or behavior. They are cameras generally fixed to a tree that take a short sequence of images when triggered. Deep learning has the potential to overcome the workload to automate image classification according to taxon or empty images. However, a standard deep neural network classifier fails because animals often represent a small portion of the high-definition images. That is why we propose a workflow named Weakly Object Detection Faster-RCNN+FPN which sui…
Animal pose estimation has recently come into the limelight due to its application in biology, zoology, and aquaculture. Deep learning methods have effectively been applied to human pose estimation. However, the major bottleneck to the application of these methods to animal pose estimation is the unavailability of sufficient quantities of labeled data. Though there are ample quantities of unlabelled data publicly available, it is economically impractical to label large quantities of data for each animal. In addition, due to the wide variety of body shapes in the animal kingdom, the transfer of…
University of Washington（华盛顿大学）；Microsoft AI for Earth（微软地球人工智能项目）
Camera Traps are extensively used to observe wildlife in their natural habitat without disturbing the ecosystem. This could help in the early detection of natural or human threats to animals, and help towards ecological conservation. Currently, a massive number of such camera traps have been deployed at various ecological conservation areas around the world, collecting data for decades, thereby requiring automation to detect images containing animals. Existing systems perform classification to detect if images contain animals by considering a single image. However, due to challenging scenes wi…
Max Planck Institute of Animal Behavior（马克斯·普朗克动物行为研究所）；Centre for the Advanced Study of Collective Behaviour, University of Konstanz（康斯坦茨大学集体行为高级研究中心）；University of Konstanz（康斯坦茨大学）；Institute for Applied Mathematics and Information Technologies, IMATI-CNR（意大利国家研究委员会应用数学与信息技术研究所）；University of Münster（明斯特大学）；Ecole Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Wageningen University（瓦赫宁根大学）；University of Bristol（布里斯托尔大学）；North Carolina State University（北卡罗来纳州立大学）；North Carolina Museum of Natural Sciences（北卡罗来纳自然科学博物馆）；Cornell Lab of Ornithology, Cornell University（康奈尔大学鸟类学实验室）；Cornell University（康奈尔大学）
Data acquisition in animal ecology is rapidly accelerating due to inexpensive and accessible sensors such as smartphones, drones, satellites, audio recorders and bio-logging devices. These new technologies and the data they generate hold great potential for large-scale environmental monitoring and understanding, but are limited by current data processing approaches which are inefficient in how they ingest, digest, and distill data into relevant information. We argue that machine learning, and especially deep learning approaches, can meet this analytic challenge to enhance our understanding, mo…
University of Bonn（波恩大学）；German Centre for Integrative Biodiversity Research (iDiv) Halle-Jena-Leipzig（德国哈勒-耶拿-莱比锡综合生物多样性研究中心（iDiv））
The development and application of modern technology is an essential basis for the efficient monitoring of species in natural habitats and landscapes to trace the development of ecosystems, species communities, and populations, and to analyze reasons of changes. For estimating animal abundance using methods such as camera trap distance sampling, spatial information of natural habitats in terms of 3D (three-dimensional) measurements is crucial. Additionally, 3D information improves the accuracy of animal detection using camera trapping. This study presents a novel approach to 3D camera trapping…
Classification and identification of wild animals for tracking and protection purposes has become increasingly important with the deterioration of the environment, and technology is the agent of change which augments this process with novel solutions. Computer vision is one such technology which uses the abilities of artificial intelligence and machine learning models on visual inputs. Convolution neural networks (CNNs) have multiple layers which have different weights for the purpose of prediction of a particular input. The precedent for classification, however, is set by the image processing…
Camera traps, unmanned observation devices, and deep learning-based image recognition systems have greatly reduced human effort in collecting and analyzing wildlife images. However, data collected via above apparatus exhibits 1) long-tailed and 2) open-ended distribution problems. To tackle the open-set long-tailed recognition problem, we propose the Temporal Flow Mask Attention Network that comprises three key building blocks: 1) an optical flow module, 2) an attention residual module, and 3) a meta-embedding classifier. We extract temporal features of sequential frames using the optical flow…
University of Guelph（圭尔夫大学）；Vector Institute for Artificial Intelligence（向量人工智能研究所）
The ability of a researcher to re-identify (re-ID) an individual animal upon re-encounter is fundamental for addressing a broad range of questions in the study of ecosystem function, community and population dynamics, and behavioural ecology. In this review, we describe a brief history of camera traps for re-ID, present a collection of computer vision feature engineering methodologies previously used for animal re-ID, provide an introduction to the underlying mechanisms of deep learning relevant to animal re-ID, highlight the success of deep learning methods for human re-ID, describe the few e…
IMATI-CNR（意大利国家研究委员会信息与远程技术研究所）；University of California, Berkeley（加州大学伯克利分校）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
We present the first method to perform automatic 3D pose, shape and texture capture of animals from images acquired in-the-wild. In particular, we focus on the problem of capturing 3D information about Grevy's zebras from a collection of images. The Grevy's zebra is one of the most endangered species in Africa, with only a few thousand individuals left. Capturing the shape and pose of these animals can provide biologists and conservationists with information about animal health and behavior. In contrast to research on human pose, shape and texture estimation, training data for endangered speci…
Camera traps are a proven tool in biology and specifically biodiversity research. However, camera traps including depth estimation are not widely deployed, despite providing valuable context about the scene and facilitating the automation of previously laborious manual ecological methods. In this study, we propose an automated camera trap-based approach to detect and identify animals using depth estimation. To detect and identify individual animals, we propose a novel method D-Mask R-CNN for the so-called instance segmentation which is a deep learning-based technique to detect and delineate ea…
ImageNet-1k is a dataset often used for benchmarking machine learning (ML) models and evaluating tasks such as image recognition and object detection. Wild animals make up 27% of ImageNet-1k but, unlike classes representing people and objects, these data have not been closely scrutinized. In the current paper, we analyze the 13,450 images from 269 classes that represent wild animals in the ImageNet-1k validation set, with the participation of expert ecologists. We find that many of the classes are ill-defined or overlapping, and that 12% of the images are incorrectly labeled, with some classes…
Texas Tech University（德克萨斯理工大学）；Texas Parks and Wildlife Department（德克萨斯州公园与野生动物管理局）
This paper introduces an automated vision system for animal detection in trail-camera images taken from a field under the administration of the Texas Parks and Wildlife Department. As traditional wildlife counting techniques are intrusive and labor intensive to conduct, trail-camera imaging is a comparatively non-intrusive method for capturing wildlife activity. However, given the large volume of images produced from trail-cameras, manual analysis of the images remains time-consuming and inefficient. We implemented a two-stage deep convolutional neural network pipeline to find animal-containin…
Ubisoft China AI & Data Lab（育碧中国AI与数据实验室）；Sichuan University（四川大学）；Ubisoft, Singapore（育碧新加坡）；INRIA Nancy Grand Est（法国国家信息与自动化研究所南锡大东区中心）
This work introduces a novel strategy for generating synthetic training data for 2D and 3D pose estimation of animals using keyframe animations. With the objective to automate the process of creating animations for wildlife, we train several 2D and 3D pose estimation models with synthetic data, and put in place an end-to-end pipeline called ZooBuilder. The pipeline takes as input a video of an animal in the wild, and generates the corresponding 2D and 3D coordinates for each joint of the animal's skeleton. With this approach, we produce motion capture data that can be used to create animations…
University of Bonn（波恩大学）；German Centre for Integrative Biodiversity Research (iDiv) Halle-Jena-Leipzig（德国综合生物多样性研究中心（iDiv）哈勒-耶拿-莱比锡）
The biodiversity crisis is still accelerating, despite increasing efforts by the international community. Estimating animal abundance is of critical importance to assess, for example, the consequences of land-use change and invasive species on community composition, or the effectiveness of conservation interventions. Various approaches have been developed to estimate abundance of unmarked animal populations. Whereas these approaches differ in methodological details, they all require the estimation of the effective area surveyed in front of a camera trap. Until now camera-to-animal distance mea…
A unified three-branch deep learning framework for automated detection and counting of wildlife in aerial imagery
School of Geospatial Engineering and Science, Sun Yat-Sen University（中山大学地理空间工程与科学学院）；Faculty of Geo-Information Science and Earth Observation, University of Twente（特温特大学地球信息科学与对地观测学院）；Boma and Badingilo National Parks, African Parks（博马和巴丁吉洛国家公园，非洲公园组织）；Department of Ecology and Evolutionary Biology, Princeton University（普林斯顿大学生态与进化生物学系）；Lolldaiga Hills Research Programme（洛尔迪加山研究项目）；School of Biological, Earth & Environmental Sciences, Environmental Research Institute, University College Cork（科克大学环境研究所生物、地球与环境科学学院）
Automated animal censuses with aerial imagery are a vital ingredient towards wildlife conservation. Recent models are generally based on deep learning and thus require vast amounts of training data. Due to their scarcity and minuscule size, annotating animals in aerial imagery is a highly tedious process. In this project, we present a methodology to reduce the amount of required training data by resorting to self-supervised pretraining. In detail, we examine a combination of recent contrastive learning methodologies like Momentum Contrast (MoCo) and Cross-Level Instance-Group Discrimination (C…
The ongoing biodiversity crisis calls for accurate estimation of animal density and abundance to identify sources of biodiversity decline and effectiveness of conservation interventions. Camera traps together with abundance estimation methods are often employed for this purpose. The necessary distances between camera and observed animals are traditionally derived in a laborious, fully manual or semi-automatic process. Both approaches require reference image material, which is both difficult to acquire and not available for existing datasets. We propose a fully automatic approach we call AUtoma…
The SA-FARI Dataset: Segment Anything in Footage of Animals for Recognition and Identification
Conservation X Labs (CXL)（Conservation X Labs）；Meta；University of Bristol（布里斯托大学）；Hasso Plattner Institute（哈索罗滕堡研究所）；University of Oviedo（奥维多大学）；Osa Conservation（Osa保护）；Senckenberg Museum of Natural History（Senckenberg自然史博物馆）；Max Planck Institute for Evolutionary Anthropology（马克斯·普朗克进化人类学研究所）；Climate Corridors（气候走廊）
Ecole Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；University of Zurich（苏黎世大学）；Wageningen University & Research（瓦赫宁根大学及研究中心）
Unmanned aerial vehicles (UAVs) offer new opportunities for wildlife monitoring, with several advantages over traditional field-based methods. They have readily been used to count birds, marine mammals and large herbivores in different environments, tasks which are routinely performed through manual counting in large collections of images. In this paper, we propose a semi-automatic system able to detect large mammals in semi-arid Savanna. It relies on an animal-detection system based on machine learning, trained with crowd-sourced annotations provided by volunteers who manually interpreted sub…
Automated animal censuses with aerial imagery are a vital ingredient towards wildlife conservation. Recent models are generally based on supervised learning and thus require vast amounts of training data. Due to their scarcity and minuscule size, annotating animals in aerial imagery is a highly tedious process. In this project, we present a methodology to reduce the amount of required training data by resorting to self-supervised pretraining. In detail, we examine a combination of recent contrastive learning methodologies like Momentum Contrast (MoCo) and Cross-Level Instance-Group Discriminat…
Computer Vision for Primate Behavior Analysis in the Wild
German Primate Center, Leibniz Institute for Primate Research（德国灵长类动物中心，莱布尼茨灵长类动物研究所）；Max Planck Institute for Dynamics and Self-Organization（马克斯·普朗克动力学与自组织研究所）
California Institute of Technology（加州理工学院）；Harvard University（哈佛大学）
To protect the location of camera trap data containing sensitive, high-target species, many ecologists randomly obfuscate the latitude and longitude of the camera when publishing their data. For example, they may publish a random location within a 1km radius of the true camera location for each camera in their network. In this paper, we investigate the robustness of geo-obfuscation for maintaining camera trap location privacy, and show via a case study that a few simple, intuitive heuristics and publicly available satellite rasters can be used to reduce the area likely to contain the camera by…
Creating high-quality articulated 3D models of animals is challenging either via manual creation or using 3D scanning tools. Therefore, techniques to reconstruct articulated 3D objects from 2D images are crucial and highly useful. In this work, we propose a practical problem setting to estimate 3D pose and shape of animals given only a few (10-30) in-the-wild images of a particular animal species (say, horse). Contrary to existing works that rely on pre-defined template shapes, we do not assume any form of 2D or 3D ground-truth annotations, nor do we leverage any multi-view or temporal informa…
Trail camera imagery has increasingly gained popularity amongst biologists for conservation and ecological research. Minimal human interference required to operate camera traps allows capturing unbiased species activities. Several studies - based on human and wildlife interactions, migratory patterns of various species, risk of extinction in endangered populations - are limited by the lack of rich data and the time-consuming nature of manually annotating trail camera imagery. We introduce a challenging wildlife camera trap classification dataset collected from two different locations in Southw…
Empowering Wildlife Guardians: An Equitable Digital Stewardship and Reward System for Biodiversity Conservation using Deep Learning and 3/4G Camera Traps
Liverpool John Moores University（利物浦约翰摩尔斯大学）；Gap Africa Projects（Gap非洲项目）；Welgevonden Game Reserve（韦尔格翁登禁猎区）；Borneo Futures（婆罗洲未来组织）；Czech Technical University（捷克技术大学）
Identifying animals from a large group of possible individuals is very important for biodiversity monitoring and especially for collecting data on a small number of particularly interesting individuals, as these have to be identified first before this can be done. Identifying them can be a very time-consuming task. This is especially true, if the animals look very similar and have only a small number of distinctive features, like elephants do. In most cases the animals stay at one place only for a short period of time during which the animal needs to be identified for knowing whether it is imp…
Knowledge over the number of animals in large wildlife reserves is a vital necessity for park rangers in their efforts to protect endangered species. Manual animal censuses are dangerous and expensive, hence Unmanned Aerial Vehicles (UAVs) with consumer level digital cameras are becoming a popular alternative tool to estimate livestock. Several works have been proposed that semi-automatically process UAV images to detect animals, of which some employ Convolutional Neural Networks (CNNs), a recent family of deep learning algorithms that proved very effective in object detection in large dataset…
