CoIn3D: Revisiting Configuration-Invariant Multi-Camera 3D Object Detection
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所人机混合增强智能全国重点实验室）；Intelligent Transportation Thrust of the Systems Hub, HKUST(GZ)（香港科技大学（广州）系统枢纽智能交通学域）；Amazon Alexa AI（亚马逊Alexa人工智能）
Multi-camera 3D object detection (MC3D) has attracted increasing attention with the growing deployment of multi-sensor physical agents, such as robots and autonomous vehicles. However, MC3D models still struggle to generalize to unseen platforms with new multi-camera configurations. Current solutions simply employ a meta-camera for unified representation but lack comprehensive consideration. In this paper, we revisit this issue and identify that the devil lies in spatial prior discrepancies across source and target configurations, including different intrinsics, extrinsics, and array layouts. To address this, we propose CoIn3D, a generalizable MC3D framework that enables strong transferability from source configurations to unseen target ones. CoIn3D explicitly incorporates all identified spatial priors into both feature embedding and image observation through spatial-aware feature modulation (SFM) and camera-aware data augmentation (CDA), respectively. SFM enriches feature space by integrating four spatial representations, such as focal length, ground depth, ground gradient, and Plücker coordinate. CDA improves observation diversity under various configurations via a training-free dynamic novel-view image synthesis scheme. Extensive experiments demonstrate that CoIn3D achieves strong cross-configuration performance on landmark datasets such as NuScenes, Waymo, and Lyft, under three dominant MC3D paradigms represented by BEVDepth, BEVFormer, and PETR.
With the recent proliferation of consumer-grade 360° cameras, it is worth revisiting visual perception challenges with spherical cameras given the potential benefit of their global field of view. To this end we introduce a spherical convolutional hourglass network (SCHN) for the dense labeling on the sphere. The SCHN is invariant to camera orientation (lifting the usual requirement for `upright' panoramic images), and its design is scalable for larger practical datasets. Initial experiments show promising results on a spherical semantic segmentation task.
Revisiting Change Detection Methods for their Application to Serac Fall Time-Lapse Monitoring
Université Lumière Lyon 2, CNRS, Ecole Centrale de Lyon, INSA Lyon, Université Claude Bernard Lyon 1, LIRIS, UMR5205（里尔大学 Lyon 2，法国国家科学研究中心，中央理工学院，里昂国立应用科学学院，里尔大学 Lyon 1，LIRIS，UMR5205）；Styx4D, 19 rue lac Saint André, Le Bourget-du-Lac, 73370, France（Styx4D，19 rue lac Saint André，Le Bourget-du-Lac，73370，法国）
The integration of multiple cameras and 3D Li- DARs has become basic configuration of augmented reality devices, robotics, and autonomous vehicles. The calibration of multi-modal sensors is crucial for a system to properly function, but it remains tedious and impractical for mass production. Moreover, most devices require re-calibration after usage for certain period of time. In this paper, we propose a single-shot solution for calibrating extrinsic transformations among multiple cameras and 3D LiDARs. We establish a panoramic infrastructure, in which a camera or LiDAR can be robustly localize…
High-resolution images enable neural networks to learn richer visual representations. However, this improved performance comes at the cost of growing computational complexity, hindering their usage in latency-sensitive applications. As not all pixels are equal, skipping computations for less-important regions offers a simple and effective measure to reduce the computation. This, however, is hard to be translated into actual speedup for CNNs since it breaks the regularity of the dense convolution workload. In this paper, we introduce SparseViT that revisits activation sparsity for recent window…
Queensland University of Technology（昆士兰科技大学）；Ford Motor Company（福特汽车公司）
6-DoF visual localization systems utilize principled approaches rooted in 3D geometry to perform accurate camera pose estimation of images to a map. Current techniques use hierarchical pipelines and learned 2D feature extractors to improve scalability and increase performance. However, despite gains in typical recall@0.25m type metrics, these systems still have limited utility for real-world applications like autonomous vehicles because of their `worst' areas of performance - the locations where they provide insufficient recall at a certain required error tolerance. Here we investigate the uti…
Tampere University of Technology（坦佩雷理工大学）；Intel Finland（英特尔芬兰公司）；Czech Technical University in Prague（布拉格捷克理工大学）
We present a statistical color constancy method that relies on novel gray pixel detection and mean shift clustering. The method, called Mean Shifted Grey Pixel -- MSGP, is based on the observation: true-gray pixels are aligned towards one single direction. Our solution is compact, easy to compute and requires no training. Experiments on two real-world benchmarks show that the proposed approach outperforms state-of-the-art methods in the camera-agnostic scenario. In the setting where the camera is known, MSGP outperforms all statistical methods.
UC San Diego（加州大学圣地亚哥分校）；ShanghaiTech University（上海科技大学）；National University of Singapore（新加坡国立大学）；Shanghai AI Laboratory（上海人工智能实验室）；Technical University of Munich（慕尼黑工业大学）
Shanghai Jiao Tong University（上海交通大学）；Tencent Robotics X Lab（腾讯Robotics X实验室）
The performance of a camera network monitoring a set of targets depends crucially on the configuration of the cameras. In this paper, we investigate the reconfiguration strategy for the parameterized camera network model, with which the sensing qualities of the multiple targets can be optimized globally and simultaneously. We first propose to use the number of pixels occupied by a unit-length object in image as a metric of the sensing quality of the object, which is determined by the parameters of the camera, such as intrinsic, extrinsic, and distortional coefficients. Then, we form a single q…
ETH Zürich（苏黎世联邦理工学院）；MPI-IS（马克斯·普朗克智能系统研究所）；UESTC（电子科技大学）；USYD（悉尼大学）；KU Leuven（鲁汶大学）
Channel (or 3D filter) pruning serves as an effective way to accelerate the inference of neural networks. There has been a flurry of algorithms that try to solve this practical problem, each being claimed effective in some ways. Yet, a benchmark to compare those algorithms directly is lacking, mainly due to the complexity of the algorithms and some custom settings such as the particular network configuration or training procedure. A fair benchmark is important for the further development of channel pruning. Meanwhile, recent investigations reveal that the channel configurations discovered by p…
University of Parma（帕尔马大学）；VisLab Srl（VisLab有限责任公司）；Ambarella Inc（安霸公司）
In this paper, a complete pipeline for image-based 3D reconstruction of urban scenarios is proposed, based on PatchMatch Multi-View Stereo (MVS). Input images are firstly fed into an off-the-shelf visual SLAM system to extract camera poses and sparse keypoints, which are used to initialize PatchMatch optimization. Then, pixelwise depths and normals are iteratively computed in a multi-scale framework with a novel depth-normal consistency loss term and a global refinement algorithm to balance the inherently local nature of PatchMatch. Finally, a large-scale point cloud is generated by back-proje…
Video-based person reID is an important task, which has received much attention in recent years due to the increasing demand in surveillance and camera networks. A typical video-based person reID system consists of three parts: an image-level feature extractor (e.g. CNN), a temporal modeling method to aggregate temporal features and a loss function. Although many methods on temporal modeling have been proposed, it is hard to directly compare these methods, because the choice of feature extractor and loss function also have a large impact on the final performance. We comprehensively study and c…
Anhui University（安徽大学）；Tencent YouTu Lab（腾讯优图实验室）；Beijing Institute of Technology（北京理工大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Peng Cheng Laboratory（鹏城实验室）；Peking University（北京大学）
The main streams of human activity recognition (HAR) algorithms are developed based on RGB cameras which are suffered from illumination, fast motion, privacy-preserving, and large energy consumption. Meanwhile, the biologically inspired event cameras attracted great interest due to their unique features, such as high dynamic range, dense temporal but sparse spatial resolution, low latency, low power, etc. As it is a newly arising sensor, even there is no realistic large-scale dataset for HAR. Considering its great practical value, in this paper, we propose a large-scale benchmark dataset to br…
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；School of Electronic and Information Engineering, Anhui University（安徽大学电子与信息工程学院）
Combining traditional RGB cameras with bio-inspired event cameras for robust object tracking has garnered increasing attention in recent years. However, most existing multimodal tracking algorithms depend heavily on high-complexity Vision Transformer architectures for feature extraction and fusion across modalities. This not only leads to substantial computational overhead but also limits the effectiveness of cross-modal interactions. In this paper, we propose an efficient RGB-Event object tracking framework based on the linear-complexity Vision Mamba network, termed Mamba-FETrack V2. Specific…
There is a requirement to determine whether a given person of interest has already been observed over a network of cameras in video surveillance systems. A human appearance obtained in one camera is usually different from the ones obtained in another camera due to difference in illumination, pose and viewpoint, camera parameters. Being related to appearance-based approaches for person re-identification, we propose a novel method based on the dominant color histogram and spatial configuration of dominant color regions on human body parts. Dominant color histogram and spatial configuration of th…
Although many advances have been made in light-field and camera-array image processing, there is still a lack of thorough analysis of the localisation accuracy of different multi-camera systems. By considering the problem from a frame-quantisation perspective, we are able to quantify the point localisation error of circular camera configurations. Specifically, we obtain closed form expressions bounding the localisation error in terms of the parameters describing the acquisition setup. These theoretical results are independent of the localisation algorithm and thus provide fundamental limits on…
Realtime and intelligent video surveillance via camera networks involve computation-intensive vision detection tasks with massive video data, which is crucial for safety in the edge-enabled industrial Internet of Things (IIoT). Multiple video streams compete for limited communication resources on the link between edge devices and camera networks, resulting in considerable communication congestion. It postpones the completion time and degrades the accuracy of vision detection tasks. Thus, achieving high accuracy of vision detection tasks under the communication constraints and vision task deadl…
University of Pennsylvania（宾夕法尼亚大学）；University of Southern California（南加利福尼亚大学）
We present a model that uses a single first-person image to generate an egocentric basketball motion sequence in the form of a 12D camera configuration trajectory, which encodes a player's 3D location and 3D head orientation throughout the sequence. To do this, we first introduce a future convolutional neural network (CNN) that predicts an initial sequence of 12D camera configurations, aiming to capture how real players move during a one-on-one basketball game. We also introduce a goal verifier network, which is trained to verify that a given camera configuration is consistent with the final g…
Florida State University（佛罗里达州立大学）；Texas Tech University（德克萨斯理工大学）
In this article, for the first time, one develops a nonparametric methodology for an analysis of shapes of configurations of landmarks on real 3D objects from regular camera photographs, thus making 3D shape analysis very accessible. A fundamental result in computer vision by Faugeras (1992), Hartley, Gupta and Chang (1992) is that generically, a finite 3D configuration of points can be retrieved up to a projective transformation, from corresponding configurations in a pair of camera images. Consequently, the projective shape of a 3D configuration can be retrieved from two of its planar views.…
We focus on better understanding the critical factors of augmentation-invariant representation learning. We revisit MoCo v2 and BYOL and try to prove the authenticity of the following assumption: different frameworks bring about representations of different characteristics even with the same pretext task. We establish the first benchmark for fair comparisons between MoCo v2 and BYOL, and observe: (i) sophisticated model configurations enable better adaptation to pre-training dataset; (ii) mismatched optimization strategies of pre-training and fine-tuning hinder model from achieving competitive…
Six-Point Method for Multi-Camera Systems with Reduced Solution Space
College of Aerospace Science and Engineering, National University of Defense Technology（国防科技大学航天科学与工程学院）；Mobile Perception Lab, ShanghaiTech University（上海科技大学移动感知实验室）
University of Oulu（奥卢大学）；Czech Technical University in Prague（布拉格捷克理工大学）；ETH Zürich（苏黎世联邦理工学院）
In this paper we study the problem of estimating the semi-generalized pose of a partially calibrated camera, i.e., the pose of a perspective camera with unknown focal length w.r.t. a generalized camera, from a hybrid set of 2D-2D and 2D-3D point correspondences. We study all possible camera configurations within the generalized camera system. To derive practical solvers to previously unsolved challenging configurations, we test different parameterizations as well as different solving strategies based on the state-of-the-art methods for generating efficient polynomial solvers. We evaluate the t…
Computer Vision Center（计算机视觉中心）；Universitat Autònoma de Barcelona（巴塞罗那自治大学）
Vision-based bronchoscopy (VB) models require the registration of the virtual lung model with the frames from the video bronchoscopy to provide effective guidance during the biopsy. The registration can be achieved by either tracking the position and orientation of the bronchoscopy camera or by calibrating its deviation from the pose (position and orientation) simulated in the virtual lung model. Recent advances in neural networks and temporal image processing have provided new opportunities for guided bronchoscopy. However, such progress has been hindered by the lack of comparative experiment…
Inria（法国国家信息与自动化研究所）；École Normale Supérieure（巴黎高等师范学院）；CNRS（法国国家科学研究中心）；PSL Research University（巴黎文理研究大学）；UC Davis（加州大学戴维斯分校）
A set of fundamental matrices relating pairs of cameras in some configuration can be represented as edges of a "viewing graph". Whether or not these fundamental matrices are generically sufficient to recover the global camera configuration depends on the structure of this graph. We study characterizations of "solvable" viewing graphs and present several new results that can be applied to determine which pairs of views may be used to recover all camera parameters. We also discuss strategies for verifying the solvability of a graph computationally.
This paper presents a novel calibration algorithm for plenoptic cameras, especially the multi-focus configuration, where several types of micro-lenses are used, using raw images only. Current calibration methods rely on simplified projection models, use features from reconstructed images, or require separated calibrations for each type of micro-lens. In the multi-focus configuration, the same part of a scene will demonstrate different amounts of blur according to the micro-lens focal length. Usually, only micro-images with the smallest amount of blur are used. In order to exploit all available…
Revisiting Color-Event based Tracking: A Unified Network, Dataset, and Metric
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Optics and Electronics, Chinese Academy of Sciences（中国科学院光电技术研究所）；Anhui University（安徽大学）；Beijing Institute of Technology（北京理工大学）；Peng Cheng Laboratory（鹏城实验室）；Peking University（北京大学）
