近年来，基于多视图图像的3D目标检测受到了广泛关注。现有方法主要从多视图图像构建3D表征，并采用密集检测头进行目标检测，或是利用分布在3D空间中的目标查询来定位目标。本文设计了多视图2D目标引导的3D目标检测器MV2D（Multi-View 2D Objects guided 3D Object Detector），可将任意2D目标检测器拓展至多视图3D目标检测任务。由于2D检测结果能为目标存在性提供有价值的先验信息，MV2D利用2D检测器生成依赖丰富图像语义的目标查询，这些动态生成的查询可帮助MV2D召回视野内的目标，展现出强大的3D目标定位能力。针对生成的查询，我们设计了稀疏交叉注意力模块，强制其聚焦于特定目标的特征，从而抑制噪声干扰。在nuScenes数据集上的评估结果表明，动态目标查询与稀疏特征聚合能够提升3D检测性能，MV2D在现有方法中达到了当前最优水平。我们希望MV2D能成为未来研究的新基准，代码可在https://github.com/tusen-ai/MV2D 获取。
英文摘要：
3D object detection from multi-view images has drawn much attention over the past few years. Existing methods mainly establish 3D representations from multi-view images and adopt a dense detection head for object detection, or employ object queries distributed in 3D space to localize objects. In this paper, we design Multi-View 2D Objects guided 3D Object Detector (MV2D), which can lift any 2D object detector to multi-view 3D object detection. Since 2D detections can provide valuable priors for object existence, MV2D exploits 2D detectors to generate object queries conditioned on the rich image semantics. These dynamically generated queries help MV2D to recall objects in the field of view and show a strong capability of localizing 3D objects. For the generated queries, we design a sparse cross attention module to force them to focus on the features of specific objects, which suppresses interference from noises. The evaluation results on the nuScenes dataset demonstrate the dynamic object queries and sparse feature aggregation can promote 3D detection capability. MV2D also exhibits a state-of-the-art performance among existing methods. We hope MV2D can serve as a new baseline for future research. Code is available at \url{https://github.com/tusen-ai/MV2D}.
Nanjing University（南京大学）；MYbank, Ant Group（蚂蚁集团网商银行）
Traditional object detectors employ the dense paradigm of scanning over locations and scales in an image. The recent query-based object detectors break this convention by decoding image features with a set of learnable queries. However, this paradigm still suffers from slow convergence, limited performance, and design complexity of extra networks between backbone and decoder. In this paper, we find that the key to these issues is the adaptability of decoders for casting queries to varying objects. Accordingly, we propose a fast-converging query-based detector, named AdaMixer, by improving the …
This paper investigates a phenomenon where query-based object detectors mispredict at the last decoding stage while predicting correctly at an intermediate stage. We review the training process and attribute the overlooked phenomenon to two limitations: lack of training emphasis and cascading errors from decoding sequence. We design and present Selective Query Recollection (SQR), a simple and effective training strategy for query-based object detectors. It cumulatively collects intermediate queries as decoding stages go deeper and selectively forwards the queries to the downstream stages aside…
Huazhong University of Science & Technology（华中科技大学）；Horizon Robotics（地平线机器人）
The query mechanism introduced in the DETR method is changing the paradigm of object detection and recently there are many query-based methods have obtained strong object detection performance. However, the current query-based detection pipelines suffer from the following two issues. Firstly, multi-stage decoders are required to optimize the randomly initialized object queries, incurring a large computation burden. Secondly, the queries are fixed after training, leading to unsatisfying generalization capability. To remedy the above issues, we present featurized object queries predicted by a qu…
In this paper, we propose a novel query design for the transformer-based object detection. In previous transformer-based detectors, the object queries are a set of learned embeddings. However, each learned embedding does not have an explicit physical meaning and we cannot explain where it will focus on. It is difficult to optimize as the prediction slot of each object query does not have a specific mode. In other words, each object query will not focus on a specific region. To solved these problems, in our query design, object queries are based on anchor points, which are widely used in CNN-ba…
DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；Nanyang Technological University（南洋理工大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
The 3D pose estimation from a single image is a challenging problem due to depth ambiguity. One type of the previous methods lifts 2D joints, obtained by resorting to external 2D pose detectors, to the 3D space. However, this type of approaches discards the contextual information of images which are strong cues for 3D pose estimation. Meanwhile, some other methods predict the joints directly from monocular images but adopt a 2.5D output representation $P^{2.5D} = (u,v,z^{r}) $ where both $u$ and $v$ are in the image space but $z^{r}$ in root-relative 3D space. Thus, the ground-truth informatio…
Google Research（谷歌研究院）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）
When interacting with objects through cameras, or pictures, users often have a specific intent. For example, they may want to perform a visual search. With most object detection models relying on image pixels as their sole input, undesired results are not uncommon. Most typically: lack of a high-confidence detection on the object of interest, or detection with a wrong class label. The issue is especially severe when operating capacity-constrained mobile object detectors on-device. In this paper we investigate techniques to modulate mobile detectors to explicitly account for the user intent, ex…
Institute of Computing Technology (ICT), Chinese Academy of Sciences（中国科学院计算技术研究所）；Stanford University（斯坦福大学）；University of Chinese Academy of Sciences（中国科学院大学）；AI Lab, Lenovo Research（联想研究院人工智能实验室）；Lenovo Ltd.（联想集团有限公司）
Recently, the dominant DETR-based approaches apply central-concept spatial prior to accelerate Transformer detector convergency. These methods gradually refine the reference points to the center of target objects and imbue object queries with the updated central reference information for spatially conditional attention. However, centralizing reference points may severely deteriorate queries' saliency and confuse detectors due to the indiscriminative spatial prior. To bridge the gap between the reference points of salient queries and Transformer detectors, we propose SAlient Point-based DETR (S…
LiDAR sensors are widely used for 3D object detection in various mobile robotics applications. LiDAR sensors continuously generate point cloud data in real-time. Conventional 3D object detectors detect objects using a set of points acquired over a fixed duration. However, recent studies have shown that the performance of object detection can be further enhanced by utilizing spatio-temporal information obtained from point cloud sequences. In this paper, we propose a new 3D object detector, named D-Align, which can effectively produce strong bird's-eye-view (BEV) features by aligning and aggrega…
University of California, Riverside（加州大学河滨分校）；Palo Alto Research Center（帕洛阿尔托研究中心）
Adversarial attacks perturb images such that a deep neural network produces incorrect classification results. A promising approach to defend against adversarial attacks on natural multi-object scenes is to impose a context-consistency check, wherein, if the detected objects are not consistent with an appropriately defined context, then an attack is suspected. Stronger attacks are needed to fool such context-aware detectors. We present the first approach for generating context-consistent adversarial attacks that can evade the context-consistency check of black-box object detectors operating on …
The Chinese University of Hong Kong（香港中文大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Although DETR-based 3D detectors can simplify the detection pipeline and achieve direct sparse predictions, their performance still lags behind dense detectors with post-processing for 3D object detection from point clouds. DETRs usually adopt a larger number of queries than GTs (e.g., 300 queries v.s. 40 objects in Waymo) in a scene, which inevitably incur many false positives during inference. In this paper, we propose a simple yet effective sparse 3D detector, named Query Contrast Voxel-DETR (ConQueR), to eliminate the challenging false positives, and achieve more accurate and sparser predi…
U-DECN: End-to-End Underwater Object Detection ConvNet with Improved DeNoising Training
National Key Laboratory of Autonomous Marine Vehicle Technology, Harbin Engineering University（自主水下机动车辆技术国家重点实验室，哈尔滨工程大学）；Hong Kong Polytechnic University（香港理工大学）
Beihang University（北京航空航天大学）；Shanghai AI Laboratory（上海人工智能实验室）；Zhejiang University（浙江大学）
Previous top-performing methods for 3D instance segmentation often maintain inter-task dependencies and the tendency towards a lack of robustness. Besides, inevitable variations of different datasets make these methods become particularly sensitive to hyper-parameter values and manifest poor generalization capability. In this paper, we address the aforementioned challenges by proposing a novel query-based method, termed as 3D-QueryIS, which is detector-free, semantic segmentation-free, and cluster-free. Specifically, we propose to generate representative points in an implicit manner, and use t…
DS-Det: Single-Query Paradigm and Attention Disentangled Learning for Flexible Object Detection
RITAS, Southern University of Science and Technology（南方科技大学RITAS）；Pengcheng Laboratory（鹏城实验室）；Pazhou Laboratory (Huangpu)（黄埔实验室）；Department of Computer Science and Engineering, Southern University of Science and Technology（南方科技大学计算机科学与工程系）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Video object detection has been an important yet challenging topic in computer vision. Traditional methods mainly focus on designing the image-level or box-level feature propagation strategies to exploit temporal information. This paper argues that with a more effective and efficient feature propagation framework, video object detectors can gain improvement in terms of both accuracy and speed. For this purpose, this paper studies object-level feature propagation, and proposes an object query propagation (QueryProp) framework for high-performance video object detection. The proposed QueryProp c…
OV-DQUO: Open-Vocabulary DETR with Denoising Text Query Training and Open-World Unknown Objects Supervision
Harbin Institute of Technology（哈尔滨工业大学）；Shenzhen University（深圳大学）；University of Chinese Academy of Sciences（中国科学院大学）；Harbin Institute of Technology, Chongqing Research Institute（哈尔滨工业大学重庆研究院）；CECloud Computing Technology Co., Ltd（天翼云计算技术有限公司）
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Inception Institute of Artificial Intelligence (IIAI)（启程人工智能研究院）
For person re-identification, existing deep networks often focus on representation learning. However, without transfer learning, the learned model is fixed as is, which is not adaptable for handling various unseen scenarios. In this paper, beyond representation learning, we consider how to formulate person image matching directly in deep feature maps. We treat image matching as finding local correspondences in feature maps, and construct query-adaptive convolution kernels on the fly to achieve local matching. In this way, the matching process and results are interpretable, and this explicit ma…
Carnegie Mellon University（卡内基梅隆大学）；Microsoft Research Asia（微软亚洲研究院）
One-shot object detection aims at detecting novel objects according to merely one given instance. With extreme data scarcity, current approaches explore various feature fusions to obtain directly transferable meta-knowledge. Yet, their performances are often unsatisfactory. In this paper, we attribute this to inappropriate correlation methods that misalign query-support semantics by overlooking spatial structures and scale variances. Upon analysis, we leverage the attention mechanism and propose a simple but effective architecture named Semantic-aligned Fusion Transformer (SaFT) to resolve the…
While general object detection with deep learning has achieved great success in the past few years, the performance and efficiency of detecting small objects are far from satisfactory. The most common and effective way to promote small object detection is to use high-resolution images or feature maps. However, both approaches induce costly computation since the computational cost grows squarely as the size of images and features increases. To get the best of two worlds, we propose QueryDet that uses a novel query mechanism to accelerate the inference speed of feature-pyramid based object detec…
Vision-based Lifting of 2D Object Detections for Automated Driving
Intel. Systems and Prod. Engineering（英特尔系统与生产工程）；FZI Research Center for Inf. Technology（弗劳恩霍夫研究所信息技术研究中心）；Institute of Meas. and Control Systems（测量与控制系统研究所）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）
CrowdQuery: Density-Guided Query Module for Enhanced 2D and 3D Detection in Crowded Scenes
Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）；Mercedes-Benz AG, Research and Development（梅赛德斯-奔驰集团，研发部门）；Intelligent Vehicles Group at TU Delft（代尔夫特理工大学智能车辆组）；Research Center for Information Technology (FZI)（信息科技研究中心）
We propose a new approach, Synthetic Optimized Layout with Instance Detection (SOLID), to pretrain object detectors with synthetic images. Our "SOLID" approach consists of two main components: (1) generating synthetic images using a collection of unlabelled 3D models with optimized scene arrangement; (2) pretraining an object detector on "instance detection" task - given a query image depicting an object, detecting all instances of the exact same object in a target image. Our approach does not need any semantic labels for pretraining and allows the use of arbitrary, diverse 3D models. Experime…
Huazhong University of Science & Technology（华中科技大学）
Recently, query based object detection frameworks achieve comparable performance with previous state-of-the-art object detectors. However, how to fully leverage such frameworks to perform instance segmentation remains an open problem. In this paper, we present QueryInst (Instances as Queries), a query based instance segmentation method driven by parallel supervision on dynamic mask heads. The key insight of QueryInst is to leverage the intrinsic one-to-one correspondence in object queries across different stages, as well as one-to-one correspondence between mask RoI features and object queries…
MEGVII Technology（旷视科技）；Shanghai Jiao Tong University（上海交通大学）；University of Hong Kong（香港大学）
In this paper, we propose a new query-based detection framework for crowd detection. Previous query-based detectors suffer from two drawbacks: first, multiple predictions will be inferred for a single object, typically in crowded scenes; second, the performance saturates as the depth of the decoding stage increases. Benefiting from the nature of the one-to-one label assignment rule, we propose a progressive predicting method to address the above issues. Specifically, we first select accepted queries prone to generate true positive predictions, then refine the rest noisy queries according to th…
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）
Few-shot object detection, expecting detectors to detect novel classes with a few instances, has made conspicuous progress. However, the prototypes extracted by existing meta-learning based methods still suffer from insufficient representative information and lack awareness of query images, which cannot be adaptively tailored to different query images. Firstly, only the support images are involved for extracting prototypes, resulting in scarce perceptual information of query images. Secondly, all pixels of all support images are treated equally when aggregating features into prototype vectors,…
Thanks to the success of object detection technology, we can retrieve objects of the specified classes even from huge image collections. However, the current state-of-the-art object detectors (such as Faster R-CNN) can only handle pre-specified classes. In addition, large amounts of positive and negative visual samples are required for training. In this paper, we address the problem of open-vocabulary object retrieval and localization, where the target object is specified by a textual query (e.g., a word or phrase). We first propose Query-Adaptive R-CNN, a simple extension of Faster R-CNN adap…
VFM$^{4}$SDG: Unveiling the Power of VFMs for Single-Domain Generalized Object Detection
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学院）；Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology（深圳先进技术大学计算机科学与人工智能学院）
Toward Accurate Camera-based 3D Object Detection via Cascade Depth Estimation and Calibration
School of Data Science（数据科学学院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（香港中文大学（深圳））；NIO（蔚来汽车）
StereoMV2D: A Sparse Temporal Stereo-Enhanced Framework for Robust Multi-View 3D Object Detection
the school of automation, Northwestern Polytechnical University（自动化学院，西北工业大学）；the school of electronic and information engineering, Suzhou University of Science and Technology（电子信息工程学院，苏州科技大学）
