Cross-view person matching and 3D human pose estimation in multi-camera networks are particularly difficult when the cameras are extrinsically uncalibrated. Existing efforts generally require large amounts of 3D data for training neural networks or known camera poses for geometric constraints to solve the problem. However, camera poses and 3D data annotation are usually expensive and not always available. We present a method, PME, that solves the two tasks without requiring either information. Our idea is to address cross-view person matching as a clustering problem using each person as a cluster center, then obtain correspondences from person matches, and estimate 3D human poses through multi-view triangulation and bundle adjustment. We solve the clustering problem by introducing a "size constraint" using the number of cameras and a "source constraint" using the fact that two people from the same camera view should not match, to narrow the solution space to a small feasible region. The 2D human poses used in clustering are obtained through a pre-trained 2D pose detector, so our method does not require expensive 3D training data for each new scene. We extensively evaluate our method on three open datasets and two indoor and outdoor datasets collected using arbitrarily set cameras. Our method outperforms other methods by a large margin on cross-view person matching, reaches SOTA performance on 3D human pose estimation without using either camera poses or 3D training data, and shows good generalization ability across five datasets of various environment settings.
We present a new color photometric stereo (CPS) method that recovers high quality, detailed 3D face geometry in a single shot. Our system uses three uncalibrated near point lights of different colors and a single camera. For robust self-calibration of the light sources, we use 3D morphable model (3DMM) and semantic segmentation of facial parts. We address the spectral ambiguity problem by incorporating albedo consensus, albedo similarity, and proxy prior into a unified framework. We avoid the need for spatial constancy of albedo; instead, we use a new measure for albedo similarity that is base…
Dynamic multi-person mesh recovery has been a hot topic in 3D vision recently. However, few works focus on the multi-person motion capture from uncalibrated cameras, which mainly faces two challenges: the one is that inter-person interactions and occlusions introduce inherent ambiguities for both camera calibration and motion capture; The other is that a lack of dense correspondences can be used to constrain sparse camera geometries in a dynamic multi-person scene. Our key idea is incorporating motion prior knowledge into simultaneous optimization of extrinsic camera parameters and human meshe…
Seoul National University（首尔国立大学）；Kwangwoon University（光云大学）
Although significant improvement has been achieved recently in 3D human pose estimation, most of the previous methods only treat a single-person case. In this work, we firstly propose a fully learning-based, camera distance-aware top-down approach for 3D multi-person pose estimation from a single RGB image. The pipeline of the proposed system consists of human detection, absolute 3D human root localization, and root-relative 3D single-person pose estimation modules. Our system achieves comparable results with the state-of-the-art 3D single-person pose estimation models without any groundtruth …
University of Michigan（密歇根大学）；Collaborative Safety Research Center at the Toyota Motor North America Research & Development（丰田汽车北美研发中心协作安全研究中心）
This paper proposes a method to extract the position and pose of vehicles in the 3D world from a single traffic camera. Most previous monocular 3D vehicle detection algorithms focused on cameras on vehicles from the perspective of a driver, and assumed known intrinsic and extrinsic calibration. On the contrary, this paper focuses on the same task using uncalibrated monocular traffic cameras. We observe that the homography between the road plane and the image plane is essential to 3D vehicle detection and the data synthesis for this task, and the homography can be estimated without the camera i…
Paris Seine University（巴黎塞纳大学）；ENSEA（国立电子与应用技术学院）；CNRS（法国国家科学研究中心）；Samsung Research Institute（三星研究院）；Université Paris Saclay（巴黎萨克雷大学）；École des Ponts（巴黎高科路桥学校）
3D human pose estimation is frequently seen as the task of estimating 3D poses relative to the root body joint. Alternatively, we propose a 3D human pose estimation method in camera coordinates, which allows effective combination of 2D annotated data and 3D poses and a straightforward multi-view generalization. To that end, we cast the problem as a view frustum space pose estimation, where absolute depth prediction and joint relative depth estimations are disentangled. Final 3D predictions are obtained in camera coordinates by the inverse camera projection. Based on this, we also present a con…
University of Michigan（密歇根大学）；Stevens Institute of Technology（史蒂文斯理工学院）
This paper addresses the problem of learning to estimate the depth of detected objects given some measurement of camera motion (e.g., from robot kinematics or vehicle odometry). We achieve this by 1) designing a recurrent neural network (DBox) that estimates the depth of objects using a generalized representation of bounding boxes and uncalibrated camera movement and 2) introducing the Object Depth via Motion and Detection Dataset (ODMD). ODMD training data are extensible and configurable, and the ODMD benchmark includes 21,600 examples across four validation and test sets. These sets include …
LISPEN（LISPEN实验室）；Arts et Métiers ParisTech（巴黎高等工艺学院）
Unlike classification, position labels cannot be assigned manually by humans. For this reason, generating supervision for precise object localization is a hard task. This paper details a method to create large datasets for 3D object localization, with real world images, using an industrial robot to generate position labels. By knowledge of the geometry of the robot, we are able to automatically synchronize the images of the two cameras and the object 3D position. We applied it to generate a screw-driver localization dataset with stereo images, using a KUKA LBR iiwa robot. This dataset could th…
University of Pennsylvania（宾夕法尼亚大学）；University of Minnesota（明尼苏达大学）；UC Berkeley（加州大学伯克利分校）；ICSI（国际计算机科学研究所）
Unlike traditional third-person cameras mounted on robots, a first-person camera, captures a person's visual sensorimotor object interactions from up close. In this paper, we study the tight interplay between our momentary visual attention and motor action with objects from a first-person camera. We propose a concept of action-objects---the objects that capture person's conscious visual (watching a TV) or tactile (taking a cup) interactions. Action-objects may be task-dependent but since many tasks share common person-object spatial configurations, action-objects exhibit a characteristic 3D sp…
With the developments of dual-lens camera modules,depth information representing the third dimension of thecaptured scenes becomes available for smartphones. It isestimated by stereo matching algorithms, taking as input thetwo views captured by dual-lens cameras at slightly differ-ent viewpoints. Depth-of-field rendering (also be referred toas synthetic defocus or bokeh) is one of the trending depth-based applications. However, to achieve fast depth estima-tion on smartphones, the stereo pairs need to be rectified inthe first place. In this paper, we propose a cost-effective so-lution to perfo…
University of Bristol（布里斯托大学）；The Australian National University（澳大利亚国立大学）；The University of Hong Kong（香港大学）
This paper addresses the problem of mirror surface reconstruction, and proposes a solution based on observing the reflections of a moving reference plane on the mirror surface. Unlike previous approaches which require tedious calibration, our method can recover the camera intrinsics, the poses of the reference plane, as well as the mirror surface from the observed reflections of the reference plane under at least three unknown distinct poses. We first show that the 3D poses of the reference plane can be estimated from the reflection correspondences established between the images and the refere…
Unconstrained Multi-view Human Pose Estimation with Algebraic Priors
Chengdu Institute of Computer Applications, Chinese Academy of Sciences, Chengdu 610213, China；School of Computer Science；Technology, University of Chinese Academy of Sciences, Beijing 101408, China；Engineering, University of Electronic Science；Centre for Artificial Intelligence；Robotics at Hong Kong Institute of Science \& Innovation, Chinese Academy of Sciences, Hong Kong 999077, China.；School of Computer Science, Sichuan University, Chengdu 610065, China
Calibration of multi-camera systems, i.e. determining the relative poses between the cameras, is a prerequisite for many tasks in computer vision and robotics. Camera calibration is typically achieved using offline methods that use checkerboard calibration targets. These methods, however, often are cumbersome and lengthy, considering that a new calibration is required each time any camera pose changes. In this work, we propose a novel, marker-free online method for the extrinsic calibration of multiple smart edge sensors, relying solely on 2D human keypoint detections that are computed locally…
MUC: Mixture of Uncalibrated Cameras for Robust 3D Human Body Reconstruction
ShanghaiTech University（上海科技大学）；Shanghai Jiao Tong University（上海交通大学）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能医疗科技有限公司）；Shanghai Clinical Research and Trial Center（上海临床研究中心）
Columbia University（哥伦比亚大学）；Robotics at Google（谷歌机器人）
In this work, we present an effective multi-view approach to closed-loop end-to-end learning of precise manipulation tasks that are 3D in nature. Our method learns to accomplish these tasks using multiple statically placed but uncalibrated RGB camera views without building an explicit 3D representation such as a pointcloud or voxel grid. This multi-camera approach achieves superior task performance on difficult stacking and insertion tasks compared to single-view baselines. Single view robotic agents struggle from occlusion and challenges in estimating relative poses between points of interest…
Learning from Synchronization: Self-Supervised Uncalibrated Multi-View Person Association in Challenging Scenes
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；ICube（ICube实验室）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg（斯特拉斯堡大学医院）
Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）
Fine-detailed reconstructions are in high demand in many applications. However, most of the existing RGB-D reconstruction methods rely on pre-calculated accurate camera poses to recover the detailed surface geometry, where the representation of a surface needs to be adapted when optimizing different quantities. In this paper, we present a novel multi-view RGB-D based reconstruction method that tackles camera pose, lighting, albedo, and surface normal estimation via the utilization of a gradient signed distance field (gradient-SDF). The proposed method formulates the image rendering process usi…
Machine Perception Research Laboratory（机器感知研究实验室）；MTA SZTAKI（匈牙利科学院计算机与自动化研究所（MTA SZTAKI））；Budapest, Hungary（匈牙利布达佩斯（注意：这是地址信息，不应作为机构提取，此处原文本中机构已完整列出，此条为冗余地址，按要求应忽略））
We aim at estimating the fundamental matrix in two views from five correspondences of rotation invariant features obtained by e.g.\ the SIFT detector. The proposed minimal solver first estimates a homography from three correspondences assuming that they are co-planar and exploiting their rotational components. Then the fundamental matrix is obtained from the homography and two additional point pairs in general position. The proposed approach, combined with robust estimators like Graph-Cut RANSAC, is superior to other state-of-the-art algorithms both in terms of accuracy and number of iteration…
HiReFF: High-Resolution Feedforward Human Reconstruction from Uncalibrated Sparse-View Video
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China（北京航空航天大学虚拟现实技术与系统国家重点实验室）；Tsinghua University, Beijing, China（清华大学）；College of Computer Science, Beijing Information Science and Technology University, Beijing, China（北京信息科技大学计算机学院）；Zhongguancun Laboratory, Beijing, China（中关村实验室）
Maria Curie-Sklodowska University（玛丽亚·居里-斯克沃多夫斯卡大学）；Warsaw University of Technology（华沙理工大学）
In this paper a method for camera pose estimation from a sequence of images is presented. The method assumes camera is calibrated (intrinsic parameters are known) which allows to decrease a number of required pairs of corresponding points compared to uncalibrated case. Our algorithm can be used as a first stage in a structure from motion stereo reconstruction system.
HGGT: Robust and Flexible 3D Hand Mesh Reconstruction from Uncalibrated Images
University of Science and Technology of China（中国科学技术大学）；Nanjing University（南京大学）；Max-Planck-Institut für Informatik（马克斯·普朗克信息学研究所）；Macau University of Science and Technology（澳门科技大学）；Hong Kong University of Science and Technology（香港科技大学）；ShanghaiTech University（上海科技大学）
Self-Supervised Uncalibrated Multi-View Video Anonymization in the Operating Room
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France（斯特拉斯堡大学，法国国家科学研究中心，法国国家卫生研究院，ICube，UMR7357，法国）；University Hospital of Strasbourg, 67000 Strasbourg, France（斯特拉斯堡大学医院，法国）
Dflow s.r.l.（3Dflow公司）；Toshiba Cambridge Research Laboratory（东芝剑桥研究实验室）；Università di Verona（维罗纳大学）；eVS s.r.l.（eVS公司）；University of Udine（乌迪内大学）
This paper addresses the structure-and-motion problem, that requires to find camera motion and 3D struc- ture from point matches. A new pipeline, dubbed Samantha, is presented, that departs from the prevailing sequential paradigm and embraces instead a hierarchical approach. This method has several advantages, like a provably lower computational complexity, which is necessary to achieve true scalability, and better error containment, leading to more stability and less drift. Moreover, a practical autocalibration procedure allows to process images without ancillary information. Experiments with…
Circular markers are planar markers which offer great performances for detection and pose estimation. For an uncalibrated camera with an unknown focal length, at least the images of at least two coplanar circles are generally required to recover their poses. Unfortunately, detecting more than one ellipse in the image must be tricky and time-consuming, especially regarding concentric circles. On the other hand, when the camera is calibrated, one circle suffices but the solution is twofold and can hardly be disambiguated. Our contribution is to put beyond this limit by dealing with the uncalibra…
National University of Singapore（新加坡国立大学）；NEC Laboratories America, Inc.（NEC美国实验室公司）；University of California, San Diego（加利福尼亚大学圣迭戈分校）
Self-calibration of camera intrinsics and radial distortion has a long history of research in the computer vision community. However, it remains rare to see real applications of such techniques to modern Simultaneous Localization And Mapping (SLAM) systems, especially in driving scenarios. In this paper, we revisit the geometric approach to this problem, and provide a theoretical proof that explicitly shows the ambiguity between radial distortion and scene depth when two-view geometry is used to self-calibrate the radial distortion. In view of such geometric degeneracy, we propose a learning a…
SPOON: Towards Coherent Compositional 3D Scene Generation from Uncalibrated Multi-view Images
The University of Hong Kong（香港大学）；Shenzhen Loop Area Institute（深圳河套学院）；The Hong Kong University of Science and Technology（香港科技大学）；Sun Yat-sen University（中山大学）；TranscEngram
Joint Person Re-identification and Camera Network Topology Inference in Multiple Cameras
Gwangju Institute of Science and Technology（光州科学技术院）；Intel Visual Computing Institute（英特尔视觉计算研究所）；Saarland Informatics Campus（萨尔兰信息学园区）；ASRI（高级软件与机器人研究所）；Seoul National University（首尔大学）
Person re-identification in large-scale multi-camera networks is a challenging task because of the spatio-temporal uncertainty and high complexity due to large numbers of cameras and people. To handle these difficulties, additional information such as camera network topology should be provided, which is also difficult to automatically estimate. In this paper, we propose a unified framework which jointly solves both person re-id and camera network topology inference problems. The proposed framework takes general multi-camera network environments into account. To effectively show the superiority…
To learn camera-view invariant features for person Re-IDentification (Re-ID), the cross-camera image pairs of each person play an important role. However, such cross-view training samples could be unavailable under the ISolated Camera Supervised (ISCS) setting, e.g., a surveillance system deployed across distant scenes. To handle this challenging problem, a new pipeline is introduced by synthesizing the cross-camera samples in the feature space for model training. Specifically, the feature encoder and generator are end-to-end optimized under a novel method, Camera-Conditioned Stable Feature Ge…
Nanyang Technological University（南洋理工大学）；New York University（纽约大学）
Existing evaluation metrics for Person Re-Identification (Person ReID) models focus on system-wide performance. However, our studies reveal weaknesses due to the uneven data distributions among cameras and different camera properties that expose the ReID system to exploitation. In this work, we raise the long-ignored ReID problem of camera performance imbalance and collect a real-world privacy-aware dataset from 38 cameras to assist the study of the imbalance issue. We propose new metrics to quantify camera performance imbalance and further propose the Adversarial Pairwise Reverse Attention (A…
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；University of Bern（伯尔尼大学）；Second Spectrum Inc.（Second Spectrum公司）
Given an image sequence featuring a portion of a sports field filmed by a moving and uncalibrated camera, such as the one of the smartphones, our goal is to compute automatically in real time the focal length and extrinsic camera parameters for each image in the sequence without using a priori knowledges of the position and orientation of the camera. To this end, we propose a novel framework that combines accurate localization and robust identification of specific keypoints in the image by using a fully convolutional deep architecture. Our algorithm exploits both the field lines and the player…
Indiana University Bloomington（印第安纳大学伯明顿分校）；University of California, Davis（加利福尼亚大学戴维斯分校）
We consider scenarios in which we wish to perform joint scene understanding, object tracking, activity recognition, and other tasks in environments in which multiple people are wearing body-worn cameras while a third-person static camera also captures the scene. To do this, we need to establish person-level correspondences across first- and third-person videos, which is challenging because the camera wearer is not visible from his/her own egocentric video, preventing the use of direct feature matching. In this paper, we propose a new semi-Siamese Convolutional Neural Network architecture to ad…
Southeast University（东南大学）；University of Wollongong（伍伦贡大学）；Nanjing University（南京大学）
In person re-identification (Re-ID), supervised methods usually need a large amount of expensive label information, while unsupervised ones are still unable to deliver satisfactory identification performance. In this paper, we introduce a novel person Re-ID task called unsupervised cross-camera person Re-ID, which only needs the within-camera (intra-camera) label information but not cross-camera (inter-camera) labels which are more expensive to obtain. In real-world applications, the intra-camera label information can be easily captured by tracking algorithms or few manual annotations. In this…
Nanjing University（南京大学）；University of Wollongong（伍伦贡大学）
In this paper, we focus on the semi-supervised person re-identification (Re-ID) case, which only has the intra-camera (within-camera) labels but not inter-camera (cross-camera) labels. In real-world applications, these intra-camera labels can be readily captured by tracking algorithms or few manual annotations, when compared with cross-camera labels. In this case, it is very difficult to explore the relationships between cross-camera persons in the training stage due to the lack of cross-camera label information. To deal with this issue, we propose a novel Progressive Cross-camera Soft-label L…
Surveillance cameras are widely applied for indoor occupancy measurement and human movement perception, which benefit for building energy management and social security. To address the challenges of limited view angle of single camera as well as lacking of inter-camera collaboration, this study presents a non-overlapping multi-camera system to enlarge the surveillance area and devotes to retrieve the same person appeared from different camera views. The system is deployed in an office building and four-day videos are collected. By training a deep convolutional neural network, the proposed syst…
Unsupervised person re-identification (ReID) is a challenging task without data annotation to guide discriminative learning. Existing methods attempt to solve this problem by clustering extracted embeddings to generate pseudo labels. However, most methods ignore the intra-class gap caused by camera style variance, and some methods are relatively complex and indirect although they try to solve the negative impact of the camera style on feature distribution. To solve this problem, we propose a camera-aware style separation and contrastive learning method (CA-UReID), which directly separates came…
In a world of pervasive cameras, public spaces are often captured from multiple perspectives by cameras of different types, both fixed and mobile. An important problem is to organize these heterogeneous collections of videos by finding connections between them, such as identifying correspondences between the people appearing in the videos and the people holding or wearing the cameras. In this paper, we wish to solve two specific problems: (1) given two or more synchronized third-person videos of a scene, produce a pixel-level segmentation of each visible person and identify corresponding peopl…
