Fitting model parameters to a set of noisy data points is a common problem in computer vision. In this work, we fit the 6D camera pose to a set of noisy correspondences between the 2D input image and a known 3D environment. We estimate these correspondences from the image using a neural network. Since the correspondences often contain outliers, we utilize a robust estimator such as Random Sample Consensus (RANSAC) or Differentiable RANSAC (DSAC) to fit the pose parameters. When the problem domain, e.g. the space of all 2D-3D correspondences, is large or ambiguous, a single network does not cover the domain well. Mixture of Experts (MoE) is a popular strategy to divide a problem domain among an ensemble of specialized networks, so called experts, where a gating network decides which expert is responsible for a given input. In this work, we introduce Expert Sample Consensus (ESAC), which integrates DSAC in a MoE. Our main technical contribution is an efficient method to train ESAC jointly and end-to-end. We demonstrate experimentally that ESAC handles two real-world problems better than competing methods, i.e. scalability and ambiguity. We apply ESAC to fitting simple geometric models to synthetic images, and to camera re-localization for difficult, real datasets.
Paris Seine University（巴黎塞纳大学）；ENSEA（国立电子与应用技术学院）；CNRS（法国国家科学研究中心）；Samsung Research Institute（三星研究院）；Université Paris Saclay（巴黎萨克雷大学）；École des Ponts（巴黎高科路桥学校）
3D human pose estimation is frequently seen as the task of estimating 3D poses relative to the root body joint. Alternatively, we propose a 3D human pose estimation method in camera coordinates, which allows effective combination of 2D annotated data and 3D poses and a straightforward multi-view generalization. To that end, we cast the problem as a view frustum space pose estimation, where absolute depth prediction and joint relative depth estimations are disentangled. Final 3D predictions are obtained in camera coordinates by the inverse camera projection. Based on this, we also present a con…
Tencent AI Lab（腾讯人工智能实验室）；Shandong University（山东大学）；Tel Aviv University（特拉维夫大学）；University College London（伦敦大学学院）；Adobe Research（奥多比研究院）；CFCS, Peking University（北京大学前沿计算研究中心）
Synthesizing novel views of dynamic humans from stationary monocular cameras is a specialized but desirable setup. This is particularly attractive as it does not require static scenes, controlled environments, or specialized capture hardware. In contrast to techniques that exploit multi-view observations, the problem of modeling a dynamic scene from a single view is significantly more under-constrained and ill-posed. In this paper, we introduce Neural Motion Consensus Flow (MoCo-Flow), a representation that models dynamic humans in stationary monocular cameras using a 4D continuous time-varian…
DiffOSeg: Omni Medical Image Segmentation via Multi-Expert Collaboration Diffusion Model
West China Biomedical Big Data Center（西昌生物医学大数据中心）；West China Hospital（西昌医院）；Sichuan University（四川大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
We present a medical crowdsourcing visual analytics platform called C{$^2$}A to visualize, classify and filter crowdsourced clinical data. More specifically, C$^2$A is used to build consensus on a clinical diagnosis by visualizing crowd responses and filtering out anomalous activity. Crowdsourcing medical applications have recently shown promise where the non-expert users (the crowd) were able to achieve accuracy similar to the medical experts. This has the potential to reduce interpretation/reading time and possibly improve accuracy by building a consensus on the findings beforehand and letti…
RC-GeoCP: Geometric Consensus for 4D Radar-Camera Collaborative Perception
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）；School of Automotive Studies, Tongji University（同济大学汽车学院）；Thrust of Artificial Intelligence, Hong Kong University of Science and Technology（香港科技大学人工智能研究所）
Label distributions in camera-trap images are highly imbalanced and long-tailed, resulting in neural networks tending to be biased towards head-classes that appear frequently. Although long-tail learning has been extremely explored to address data imbalances, few studies have been conducted to consider camera-trap characteristics, such as multi-domain and multi-frame setup. Here, we propose a unified framework and introduce two datasets for long-tailed camera-trap recognition. We first design domain experts, where each expert learns to balance imperfect decision boundaries caused by data imbal…
CARE: Class-Adaptive Expert Consensus for Reliable Learning with Long-Tailed Noisy Labels
College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China.（深圳大学计算机科学与软件工程学院，中国深圳）；Guangming Laboratory, Shenzhen, China.（深圳广明实验室，中国深圳）；School of Informatics, Xiamen University, Xiamen, China.（厦门大学信息学院，中国厦门）；School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China（广东工业大学计算机科学与技术学院，中国广州）
Medical image segmentation requires consensus ground truth segmentations to be derived from multiple expert annotations. A novel approach is proposed that obtains consensus segmentations from experts using graph cuts (GC) and semi supervised learning (SSL). Popular approaches use iterative Expectation Maximization (EM) to estimate the final annotation and quantify annotator's performance. Such techniques pose the risk of getting trapped in local minima. We propose a self consistency (SC) score to quantify annotator consistency using low level image features. SSL is used to predict missing anno…
EPFL（洛桑联邦理工学院）；UBC（不列颠哥伦比亚大学）；Balgrist University Hospital（巴尔格里斯特大学医院）；ClearSpace SA（ClearSpace公司）
Self-supervised detection and segmentation of foreground objects aims for accuracy without annotated training data. However, existing approaches predominantly rely on restrictive assumptions on appearance and motion. For scenes with dynamic activities and camera motion, we propose a multi-camera framework in which geometric constraints are embedded in the form of multi-view consistency during training via coarse 3D localization in a voxel grid and fine-grained offset regression. In this manner, we learn a joint distribution of proposals over multiple views. At inference time, our method operat…
Person re-identification aims to associate images of the same person over multiple non-overlapping camera views at different times. Depending on the human operator, manual re-identification in large camera networks is highly time consuming and erroneous. Automated person re-identification is required due to the extensive quantity of visual data produced by rapid inflation of large scale distributed multi-camera systems. The state-of-the-art works focus on learning and factorize person appearance features into latent discriminative factors at multiple semantic levels. We propose Deep Parallel F…
University of Waterloo（滑铁卢大学）；Sunnybrook Health Sciences Centre（森尼布鲁克健康科学中心）
Marking tumors and organs is a challenging task suffering from both inter- and intra-observer variability. The literature quantifies observer variability by generating consensus among multiple experts when they mark the same image. Automatically building consensus contours to establish quality assurance for image segmentation is presently absent in the clinical practice. As the \emph{big data} becomes more and more available, techniques to access a large number of existing segments of multiple experts becomes possible. Fast algorithms are, hence, required to facilitate the search for similar c…
Expert Consensus on Criteria for the Automated Assessment of Laparoscopic Camera Navigation
University Medical Center Göttingen（哥廷根大学医学中心）；Fraunhofer Research Institution for Individualized and Cell-based Medical Engineering (IMTE)（弗劳恩霍夫个性化与细胞医学工程研究所）
SemLT3D: Semantic-Guided Expert Distillation for Camera-only Long-Tailed 3D Object Detection
AICV Lab, University of Arkansas, USA（AICV实验室，阿肯色大学，美国）；University of Utah, USA（犹他大学，美国）；University of Houston, USA（休斯顿大学，美国）；University of Liverpool, UK（利物浦大学，英国）
Tencent Healthcare, Tencent（腾讯医疗健康，腾讯）；Beijing Tongren Hospital, Capital Medical University（首都医科大学附属北京同仁医院）
Medical images are generally labeled by multiple experts before the final ground-truth labels are determined. Consensus or disagreement among experts regarding individual images reflects the gradeability and difficulty levels of the image. However, when being used for model training, only the final ground-truth label is utilized, while the critical information contained in the raw multi-rater gradings regarding the image being an easy/hard case is discarded. In this paper, we aim to take advantage of the raw multi-rater gradings to improve the deep learning model performance for the glaucoma c…
China Agricultural University（中国农业大学）；Key Laboratory of Agricultural Information Acquisition Technology, Ministry of Agriculture（农业部农业信息获取技术重点实验室）；Beijing Engineering and Technology Research Center for Internet of Things in Agriculture（北京农业物联网工程技术研究中心）
Aerators are essential and crucial auxiliary devices in intensive culture, especially in industrial culture in China. The traditional methods cannot accurately detect abnormal condition of aerators in time. Surveillance cameras are widely used as visual perception modules of the Internet of Things, and then using these widely existing surveillance cameras to realize real-time anomaly detection of aerators is a cost-free and easy-to-promote method. However, it is difficult to develop such an expert system due to some technical and applied challenges, e.g., illumination, occlusion, complex backg…
Expert Consensus-based Video-Based Assessment Tool for Workflow Analysis in Minimally Invasive Colorectal Surgery: Development and Validation of ColoWorkflow
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France（斯特拉斯堡大学，CNRS，INSERM，ICube，UMR7357，法国斯特拉斯堡）；Department of Medical-Surgical Sciences and Translational Medicine, Faculty of Medicine and Psychology, Sapienza University of Rome, Italy（医学与外科科学与转化医学系，医学与心理学学院，罗马萨皮恩扎大学，意大利）；University Hospital Virgen Macarena, University of Sevilla, Seville, Spain（圣地亚哥医院，塞维利亚大学，西班牙塞维利亚）；Department of General and Minimally Invasive Surgery, Fondazione IRCCS Ca' Granda Ospedale Maggiore Policlinico, Milan, Italy（普通外科与微创外科部门，IRCCS卡格拉奥斯皮奇亚诺医院，米兰，意大利）；Arizona State University, Tempe, AZ, USA（亚利桑那州立大学，Tempe，美国亚利桑那）
University of Sheffield（谢菲尔德大学）；University College of Nabi Akram（纳比·阿克拉姆大学学院）
This paper presents a state-of-the-art approach in object detection for being applied in future SLAM problems. Although, many SLAM methods are proposed to create suitable autonomy for mobile robots namely ground vehicles, they still face overconfidence and large computations during entrance to immense spaces with many landmarks. In particular, they suffer from impractical applications via sole reliance on the limited sensors like camera. Proposed method claims that unmanned ground vehicles without having huge amount of database for object definition and highly advance prediction parameters can…
In this paper, we formulate a generic non-minimal solver using the existing tools of Polynomials Optimization Problems (POP) from computational algebraic geometry. The proposed method exploits the well known Shor's or Lasserre's relaxations, whose theoretical aspects are also discussed. Notably, we further exploit the POP formulation of non-minimal solver also for the generic consensus maximization problems in 3D vision. Our framework is simple and straightforward to implement, which is also supported by three diverse applications in 3D vision, namely rigid body transformation estimation, Non-…
Beijing University of Aeronautics and Astronautics（北京航空航天大学）
One of the fundamental requirements for visual surveillance using smart camera networks is the correct association of each persons observations generated on different cameras. Recently, distributed data association that involves only local information processing on each camera node and mutual information exchanging between neighboring cameras has attracted many research interests due to its superiority in large scale applications. In this paper, we formulate the problem of data association in smart camera networks as an Integer Programming problem by introducing a set of linking variables, and…
As 360° cameras become prevalent in many autonomous systems (e.g., self-driving cars and drones), efficient 360° perception becomes more and more important. We propose a novel self-supervised learning approach for predicting the omnidirectional depth and camera motion from a 360° video. In particular, starting from the SfMLearner, which is designed for cameras with normal field-of-view, we introduce three key features to process 360° images efficiently. Firstly, we convert each image from equirectangular projection to cubic projection in order to avoid image distortion. In each network layer, …
This paper presents an algorithm for indoor layout estimation and reconstruction through the fusion of a sequence of captured images and LiDAR data sets. In the proposed system, a movable platform collects both intensity images and 2D LiDAR information. Pose estimation and semantic segmentation is computed jointly by aligning the LiDAR points to line segments from the images. For indoor scenes with walls orthogonal to floor, the alignment problem is decoupled into top-down view projection and a 2D similarity transformation estimation and solved by the recursive random sample consensus (R-RANSA…
Longitudinal Vestibular Schwannoma Dataset with Consensus-based Human-in-the-loop Annotations
School of Biomedical Engineering and Imaging Sciences, King’s College London（生物医学工程与成像科学学院，伦敦国王学院）；King’s College Hospital NHS Foundation Trust（伦敦国王学院医院国家健康服务信托基金）；The Walton Centre NHS Foundation Trust（沃尔顿中心国家健康服务信托基金）
hutom；Yonsei University College of Medicine（延世大学医学院）
As the area of application of deep neural networks expands to areas requiring expertise, e.g., in medicine and law, more exquisite annotation processes for expert knowledge training are required. In particular, it is difficult to guarantee generalization performance in the clinical field in the case of expert knowledge training where opinions may differ even among experts on annotations. To raise the issue of the annotation generation process for expertise training of CNNs, we verified the annotations for surgical phase recognition of laparoscopic cholecystectomy and subtotal gastrectomy for g…
University of Campinas（坎皮纳斯大学）；University of Calgary（卡尔加里大学）；Alberta Health Services（阿尔伯塔卫生服务局）
Convolutional neural networks (CNN) for medical imaging are constrained by the number of annotated data required in the training stage. Usually, manual annotation is considered to be the "gold standard". However, medical imaging datasets that include expert manual segmentation are scarce as this step is time-consuming, and therefore expensive. Moreover, single-rater manual annotation is most often used in data-driven approaches making the network optimal with respect to only that single expert. In this work, we propose a CNN for brain extraction in magnetic resonance (MR) imaging, that is full…
Expert-aware uncertainty estimation for quality control of neural-based blood typing
Institute for Information Transmission Problems（信息传输问题研究所）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；The Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）；AIRI（AIRI（人工智能研究机构））
This paper presents a novel indoor layout estimation system based on the fusion of 2D LiDAR and intensity camera data. A ground robot explores an indoor space with a single floor and vertical walls, and collects a sequence of intensity images and 2D LiDAR datasets. The LiDAR provides accurate depth information, while the camera captures high-resolution data for semantic interpretation. The alignment of sensor outputs and image segmentation are computed jointly by aligning LiDAR points, as samples of the room contour, to ground-wall boundaries in the images. The alignment problem is decoupled i…
Universidad de Antioquia UdeA（安蒂奥基亚大学（UdeA））；Instituto Tecnológico Metropolitano ITM（大都会理工学院（ITM））
Non intrusive monitoring of animals in the wild is possible using camera trapping framework, which uses cameras triggered by sensors to take a burst of images of animals in their habitat. However camera trapping framework produces a high volume of data (in the order on thousands or millions of images), which must be analyzed by a human expert. In this work, a method for animal species identification in the wild using very deep convolutional neural networks is presented. Multiple versions of the Snapshot Serengeti dataset were used in order to probe the ability of the method to cope with differ…
Northwestern University（西北大学）；University of California, Riverside（加州大学河滨分校）
In many intelligent systems, a network of agents collaboratively perceives the environment for better and more efficient situation awareness. As these agents often have limited resources, it could be greatly beneficial to identify the content overlapping among camera views from different agents and leverage it for reducing the processing, transmission and storage of redundant/unimportant video frames. This paper presents a consensus-based distributed multi-agent video fast-forwarding framework, named DMVF, that fast-forwards multi-view video streams collaboratively and adaptively. In our frame…
The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）；École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Qing Yuan Research Institute, Shanghai Jiao Tong University（上海交通大学清源研究院）
Correspondence selection aims to correctly select the consistent matches (inliers) from an initial set of putative correspondences. The selection is challenging since putative matches are typically extremely unbalanced, largely dominated by outliers, and the random distribution of such outliers further complicates the learning process for learning-based methods. To address this issue, we propose to progressively prune the correspondences via a local-to-global consensus learning procedure. We introduce a ``pruning'' block that lets us identify reliable candidates among the initial matches accor…
Most recent works on multi-target tracking with multiple cameras focus on centralized systems. In contrast, this paper presents a multi-target tracking approach implemented in a distributed camera network. The advantages of distributed systems lie in lighter communication management, greater robustness to failures and local decision making. On the other hand, data association and information fusion are more challenging than in a centralized setup, mostly due to the lack of global and complete information. The proposed algorithm boosts the benefits of the Distributed-Consensus Kalman Filter wit…
LTCI, Télécom Paris, Institut polytechnique de Paris（法国巴黎高等电信学院LTCI实验室、法国巴黎综合理工学院（巴黎高等电信学院））；DXOMARK（DXOMARK公司）；Télécom Paris（巴黎高等电信学院）；Institut polytechnique de Paris（巴黎综合理工学院）；LTCI（LTCI实验室）；DXOMARK Boulogne-Billancourt（DXOMARK公司布洛涅-比扬古分部）
In this work, we tackle the problem of estimating a camera capability to preserve fine texture details at a given lighting condition. Importantly, our texture preservation measurement should coincide with human perception. Consequently, we formulate our problem as a regression one and we introduce a deep convolutional network to estimate texture quality score. At training time, we use ground-truth quality scores provided by expert human annotators in order to obtain a subjective quality measure. In addition, we propose a region selection method to identify the image regions that are better sui…
We introduce a camera pipeline for rendering visually pleasing photographs in low light conditions, as part of the NTIRE2022 Night Photography Rendering challenge. Given the nature of the task, where the objective is verbally defined by an expert photographer instead of relying on explicit ground truth images, we design an handcrafted solution, characterized by a shallow structure and by a low parameter count. Our pipeline exploits a local light enhancer as a form of high dynamic range correction, followed by a global adjustment of the image histogram to prevent washed-out results. We proporti…
City University of Hong Kong（香港城市大学）；Shenzhen Research Institute, City University of Hong Kong（香港城市大学深圳研究院）；Nanyang Technological University（南洋理工大学）
There has been an increasing consensus in learning based face anti-spoofing that the divergence in terms of camera models is causing a large domain gap in real application scenarios. We describe a framework that eliminates the influence of inherent variance from acquisition cameras at the feature level, leading to the generalized face spoofing detection model that could be highly adaptive to different acquisition devices. In particular, the framework is composed of two branches. The first branch aims to learn the camera invariant spoofing features via feature level decomposition in the high fr…
Multimodal Foundation Models for Zero-shot Animal Species Recognition in Camera Trap Images
Microsoft AI for Good Lab, Redmond（微软人工智能公益实验室）；University of Southern California, Los Angeles（南加州大学）；Microsoft Research, Redmond（微软研究院）；Nanyang Technological University, Singapore（南洋理工大学）；Universidad de Los Andes, Colombia（洛斯安第斯大学）
Depth adjustment aims to enhance the visual experience of stereoscopic 3D (S3D) images, which accompanied with improving visual comfort and depth perception. For a human expert, the depth adjustment procedure is a sequence of iterative decision making. The human expert iteratively adjusts the depth until he is satisfied with the both levels of visual comfort and the perceived depth. In this work, we present a novel deep reinforcement learning (DRL)-based approach for depth adjustment named VCA-RL (Visual Comfort Aware Reinforcement Learning) to explicitly model human sequential decision making…
Smartphone is the most successful consumer electronic product in today's mobile social network era. The smartphone camera quality and its image post-processing capability is the dominant factor that impacts consumer's buying decision. However, the quality evaluation of photos taken from smartphones remains a labor-intensive work and relies on professional photographers and experts. As an extension of the prior CNN-based NR-IQA approach, we propose a multi-task deep CNN model with scene type detection as an auxiliary task. With the shared model parameters in the convolution layer, the learned f…
