Automated capture of animal pose is transforming how we study neuroscience and social behavior. Movements carry important social cues, but current methods are not able to robustly estimate pose and shape of animals, particularly for social animals such as birds, which are often occluded by each other and objects in the environment. To address this problem, we first introduce a model and multi-view optimization approach, which we use to capture the unique shape and pose space displayed by live birds. We then introduce a pipeline and experiments for keypoint, mask, pose, and shape regression that recovers accurate avian postures from single views. Finally, we provide extensive multi-view keypoint and mask annotations collected from a group of 15 social birds housed together in an outdoor aviary. The project website with videos, results, code, mesh model, and the Penn Aviary Dataset can be found at https://marcbadger.github.io/avian-mesh.
We introduce view birdification, the problem of recovering ground-plane movements of people in a crowd from an ego-centric video captured from an observer (e.g., a person or a vehicle) also moving in the crowd. Recovered ground-plane movements would provide a sound basis for situational understanding and benefit downstream applications in computer vision and robotics. In this paper, we formulate view birdification as a geometric trajectory reconstruction problem and derive a cascaded optimization method from a Bayesian perspective. The method first estimates the observer's movement and then lo…
University of Science and Technology of China（中国科学技术大学）；Cardiff University（卡迪夫大学）
Recently, 3D face reconstruction from a single image has achieved great success with the help of deep learning and shape prior knowledge, but they often fail to produce accurate geometry details. On the other hand, photometric stereo methods can recover reliable geometry details, but require dense inputs and need to solve a complex optimization problem. In this paper, we present a lightweight strategy that only requires sparse inputs or even a single image to recover high-fidelity face shapes with images captured under near-field lights. To this end, we construct a dataset containing 84 differ…
University of Alberta（阿尔伯塔大学）；Demant A/S（戴蒙特公司）；Canadian Forest Service, Natural Resources Canada（加拿大自然资源部加拿大林务局）
In nature, the collective behavior of animals, such as flying birds is dominated by the interactions between individuals of the same species. However, the study of such behavior among the bird species is a complex process that humans cannot perform using conventional visual observational techniques such as focal sampling in nature. For social animals such as birds, the mechanism of group formation can help ecologists understand the relationship between social cues and their visual characteristics over time (e.g., pose and shape). But, recovering the varying pose and shapes of flying birds is a…
Yangzhou University（扬州大学）；Institut Supérieur d'Électronique de Paris（巴黎高等电子学院）；Sorbonne University（索邦大学）；Polytechnique Montreal（蒙特利尔理工学院）；Westlake University（西湖大学）
Closed-loop architecture is widely utilized in automatic control systems and attain distinguished performance. However, classical compressive sensing systems employ open-loop architecture with separated sampling and reconstruction units. Therefore, a method of iterative compensation recovery for image compressive sensing (ICRICS) is proposed by introducing closed-loop framework into traditional compresses sensing systems. The proposed method depends on any existing approaches and upgrades their reconstruction performance by adding negative feedback structure. Theory analysis on negative feedba…
Animals are diverse in shape, but building a deformable shape model for a new species is not always possible due to the lack of 3D data. We present a method to capture new species using an articulated template and images of that species. In this work, we focus mainly on birds. Although birds represent almost twice the number of species as mammals, no accurate shape model is available. To capture a novel species, we first fit the articulated template to each training sample. By disentangling pose and shape, we learn a shape space that captures variation both among species and within each specie…
Nonperiodic dynamic CT reconstruction using backward-warping INR with regularization of diffeomorphism (BIRD)
Department of Engineering Physics, Tsinghua University（清华大学工程物理系）；Department of Radiology, Beijing Tsinghua Changgung Hospital, School of Clinical Medicine, Tsinghua Medicine, Tsinghua University（北京清华长庚医院放射科，清华大学临床医学院，清华大学医学院，清华大学）；United Imaging Healthcare, North America（联合影像（北美））；Shanghai United Imaging Healthcare Co., Ltd.（上海联合影像医疗科技有限公司）
The ability to capture detailed interactions among individuals in a social group is foundational to our study of animal behavior and neuroscience. Recent advances in deep learning and computer vision are driving rapid progress in methods that can record the actions and interactions of multiple individuals simultaneously. Many social species, such as birds, however, live deeply embedded in a three-dimensional world. This world introduces additional perceptual challenges such as occlusions, orientation-dependent appearance, large variation in apparent size, and poor sensor coverage for 3D recons…
Indian Institute of Technology, Madras（印度理工学院马德拉斯分校）
Reconstruction of signals from compressively sensed measurements is an ill-posed problem. In this paper, we leverage the recurrent generative model, RIDE, as an image prior for compressive image reconstruction. Recurrent networks can model long-range dependencies in images and hence are suitable to handle global multiplexing in reconstruction from compressive imaging. We perform MAP inference with RIDE using back-propagation to the inputs and projected gradient method. We propose an entropy thresholding based approach for preserving texture in images well. Our approach shows superior reconstru…
In this paper, we present BirdSLAM, a novel simultaneous localization and mapping (SLAM) system for the challenging scenario of autonomous driving platforms equipped with only a monocular camera. BirdSLAM tackles challenges faced by other monocular SLAM systems (such as scale ambiguity in monocular reconstruction, dynamic object localization, and uncertainty in feature representation) by using an orthographic (bird's-eye) view as the configuration space in which localization and mapping are performed. By assuming only the height of the ego-camera above the ground, BirdSLAM leverages single-vie…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of California, Berkeley（加州大学伯克利分校）
Early work in computer vision considered a host of geometric cues for both shape reconstruction and recognition. However, since then, the vision community has focused heavily on shading cues for reconstruction, and moved towards data-driven approaches for recognition. In this paper, we reconsider these perhaps overlooked "boundary" cues (such as self occlusions and folds in a surface), as well as many other established constraints for shape reconstruction. In a variety of user studies and quantitative tasks, we evaluate how well these cues inform shape reconstruction (relative to each other) i…
We introduce a framework for the recovery of points on a smooth surface in high-dimensional space, with application to dynamic imaging. We assume the surface to be the zero-level set of a bandlimited function. We show that the exponential maps of the points on the surface satisfy annihilation relations, implying that they lie in a finite dimensional subspace. We rely on nuclear norm minimization of the maps to recover the points from noisy and undersampled measurements. Since this direct approach suffers from the curse of dimensionality, we introduce an iterative reweighted algorithm that uses…
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
We show that by extending the Laplacian formalism, which was first introduced in the Graphics community to regularize 3D meshes, we can turn the monocular 3D shape reconstruction of a deformable surface given correspondences with a reference image into a much better-posed problem. This allows us to quickly and reliably eliminate outliers by simply solving a linear least squares problem. This yields an initial 3D shape estimate, which is not necessarily accurate, but whose 2D projections are. The initial shape is then refined by a constrained optimization problem to output the final surface rec…
In this paper, we introduce 3D-GMNet, a deep neural network for 3D object shape reconstruction from a single image. As the name suggests, 3D-GMNet recovers 3D shape as a Gaussian mixture. In contrast to voxels, point clouds, or meshes, a Gaussian mixture representation provides an analytical expression with a small memory footprint while accurately representing the target 3D shape. At the same time, it offers a number of additional advantages including instant pose estimation and controllable level-of-detail reconstruction, while also enabling interpretation as a point cloud, volume, and a mes…
MuNet: A Mutualistic Network for Joint 3D Human Mesh Recovery and 3D Clothed Human Reconstruction from Single Images
National Engineering Research Center for E-Learning（教育信息化国家级工程研究中心）；National Engineering Research Center of Educational Big Data（教育大数据国家级工程研究中心）；School of Electronic Information and Communications（电子信息与通讯学院）；School of Artificial Intelligence and Automation（人工智能与自动化学院）
This thesis proposes spatio-spectral techniques for hyperspectral image analysis. Adaptive spatio-spectral support and variable exposure hyperspectral imaging is demonstrated to improve spectral reflectance recovery from hyperspectral images. Novel spectral dimensionality reduction techniques have been proposed from the perspective of spectral only and spatio-spectral information preservation. It was found that the joint sparse and joint group sparse hyperspectral image models achieve lower reconstruction error and higher recognition accuracy using only a small subset of bands. Hyperspectral i…
Institute for Computer Science and Control, Hungarian Academy of Sciences（匈牙利科学院计算机科学与控制研究所）
In this paper, we propose a novel image denoising algorithm using collaborative support-agnostic sparse reconstruction. An observed image is first divided into patches. Similarly structured patches are grouped together to be utilized for collaborative processing. In the proposed collaborative schemes, similar patches are assumed to share the same support taps. For sparse reconstruction, the likelihood of a tap being active in a patch is computed and refined through a collaboration process with other similar patches in the same group. This provides very good patch support estimation, hence enha…
This paper investigates the possibility of reconstruction of images considering that they are sparse in the DCT transformation domain. Two approaches are considered. One when the image is pre-processed in the DCT domain, using 8x8 blocks. The image is made sparse by setting the smallest DCT coefficients to zero. In the other case the original image is considered without pre-processing, assuming the sparsity as intrinsic property of the analyzed image. A gradient based algorithm is used to recover a large number of missing pixels in the image. The case of a salt-and-paper noise affecting a larg…
3D recovery from multi-stereo and stereo images, as an important application of the image-based perspective geometry, serves many applications in computer vision, remote sensing and Geomatics. In this chapter, the authors utilize the imaging geometry and present approaches that perform 3D reconstruction from cross-view images that are drastically different in their viewpoints. We introduce our framework that takes ground-view images and satellite images for full 3D recovery, which includes necessary methods in satellite and ground-based point cloud generation from images, 3D data co-registrati…
We propose the Canonical 3D Deformer Map, a new representation of the 3D shape of common object categories that can be learned from a collection of 2D images of independent objects. Our method builds in a novel way on concepts from parametric deformation models, non-parametric 3D reconstruction, and canonical embeddings, combining their individual advantages. In particular, it learns to associate each image pixel with a deformation model of the corresponding 3D object point which is canonical, i.e. intrinsic to the identity of the point and shared across objects of the category. The result is …
MFSR: Multi-fractal Feature for Super-resolution Reconstruction with Fine Details Recovery
College of Sciences, Northeastern University（东北大学理学院）；School of Computer Science and Engineering, Nanjing University of Science（南京理工大学计算机科学与工程学院）；College of Renewable Energy, Hohai University（河海大学可再生能源学院）
HHMR: Holistic Hand Mesh Recovery by Enhancing the Multimodal Controllability of Graph Diffusion Models
Tsinghua University（清华大学）；Beijing Normal University（北京师范大学）；Beijing National Research Center for Information Science and Technology, Tsinghua University（北京信息科学与技术国家研究中心（清华大学））
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；California Nano Systems Institute (CNSI)（加州纳米系统研究所）；David Geffen School of Medicine, University of California, Los Angeles（加利福尼亚大学洛杉矶分校大卫·格芬医学院）
Digital holography is one of the most widely used label-free microscopy techniques in biomedical imaging. Recovery of the missing phase information of a hologram is an important step in holographic image reconstruction. Here we demonstrate a convolutional recurrent neural network (RNN) based phase recovery approach that uses multiple holograms, captured at different sample-to-sensor distances to rapidly reconstruct the phase and amplitude information of a sample, while also performing autofocusing through the same network. We demonstrated the success of this deep learning-enabled holography me…
Continuous K-space Recovery Network with Image Guidance for Fast MRI Reconstruction
The Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention（上海医学影像计算与计算机辅助介入重点实验室）；Academy of Engineering and Technology, Fudan University（复旦大学工程与技术学院）
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；California NanoSystems Institute (CNSI)（加州纳米系统研究所）；David Geffen School of Medicine（戴维·格芬医学院）
Holography encodes the three dimensional (3D) information of a sample in the form of an intensity-only recording. However, to decode the original sample image from its hologram(s), auto-focusing and phase-recovery are needed, which are in general cumbersome and time-consuming to digitally perform. Here we demonstrate a convolutional neural network (CNN) based approach that simultaneously performs auto-focusing and phase-recovery to significantly extend the depth-of-field (DOF) in holographic image reconstruction. For this, a CNN is trained by using pairs of randomly de-focused back-propagated …
University of California, Los Angeles（加州大学洛杉矶分校）；California NanoSystems Institute（加利福尼亚纳米系统研究所）；David Geffen School of Medicine, University of California, Los Angeles（加州大学洛杉矶分校大卫·格芬医学院）
Phase recovery from intensity-only measurements forms the heart of coherent imaging techniques and holography. Here we demonstrate that a neural network can learn to perform phase recovery and holographic image reconstruction after appropriate training. This deep learning-based approach provides an entirely new framework to conduct holographic imaging by rapidly eliminating twin-image and self-interference related spatial artifacts. Compared to existing approaches, this neural network based method is significantly faster to compute, and reconstructs improved phase and amplitude images of the o…
Generative priors have been shown to provide improved results over sparsity priors in linear inverse problems. However, current state of the art methods suffer from one or more of the following drawbacks: (a) speed of recovery is slow; (b) reconstruction quality is deficient; (c) reconstruction quality is contingent on a computationally expensive process of tuning hyperparameters. In this work, we address these issues by utilizing Denoising Auto Encoders (DAEs) as priors and a projected gradient descent algorithm for recovering the original signal. We provide rigorous theoretical guarantees fo…
Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）；the Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；School of Science and Engineering, the Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）理工学院）；University of Edinburgh（爱丁堡大学）；The Alan Turing Institute（艾伦·图灵研究所）
Dynamic environments that include unstructured moving objects pose a hard problem for Simultaneous Localization and Mapping (SLAM) performance. The motion of rigid objects can be typically tracked by exploiting their texture and geometric features. However, humans moving in the scene are often one of the most important, interactive targets - they are very hard to track and reconstruct robustly due to non-rigid shapes. In this work, we present a fast, learning-based human object detector to isolate the dynamic human objects and realise a real-time dense background reconstruction framework. We g…
We introduce a framework for recovering an image from its rotationally and translationally invariant features based on autocorrelation analysis. This work is an instance of the multi-target detection statistical model, which is mainly used to study the mathematical and computational properties of single-particle reconstruction using cryo-electron microscopy (cryo-EM) at low signal-to-noise ratios. We demonstrate with synthetic numerical experiments that an image can be reconstructed from rotationally and translationally invariant features and show that the reconstruction is robust to noise. Th…
Deep learning models have significantly improved the visual quality and accuracy on compressive sensing recovery. In this paper, we propose an algorithm for signal reconstruction from compressed measurements with image priors captured by a generative model. We search and constrain on latent variable space to make the method stable when the number of compressed measurements is extremely limited. We show that, by exploiting certain structures of the latent variables, the proposed method produces improved reconstruction accuracy and preserves realistic and non-smooth features in the image. Our al…
In this paper we address the problem of visual quality of images reconstructed from block-wise random projections. Independent reconstruction of the blocks can severely affect visual quality, by displaying artifacts along block borders. We propose a method to enforce smoothness across block borders by modifying the sensing and reconstruction process so as to employ partially overlapping blocks. The proposed algorithm accomplishes this by computing a fast preview from the blocks, whose purpose is twofold. On one hand, it allows to enforce a set of constraints to drive the reconstruction algorit…
