SGCCNet: Single-Stage 3D Object Detector With Saliency-Guided Data Augmentation and Confidence Correction Mechanism
Chinese Academy of Sciences（中国科学院）；Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Liaoning Province（辽宁省）
基于点的单阶段3D目标检测器因其轻量化和快速推理速度的优势而引起了广泛的研究兴趣。然而，它们仍然面临低质量目标学习不足（ILQ）以及定位精度与分类置信度之间不一致（MLC）等挑战。在本文中，我们提出SGCCNet来缓解这两个问题。针对ILQ，SGCCNet采用显著性引导数据增强（SGDA）策略，通过降低模型对显著性特征的依赖来增强其对低质量目标的鲁棒性。具体而言，我们构建了一个分类任务，然后通过在一个可微过程中将点移向点云质心来近似点的显著性分数。在训练过程中，SGCCNet将通过丢弃点的方式被迫从低显著性特征中学习。同时，为了避免因丢弃点引起的内部协变量偏移和上下文特征遗忘，我们在每个阶段添加了几何归一化模块和跳跃连接块。针对MLC，我们专门为基于点的多类别检测器设计了一种置信度校正机制（CCM）。该机制在后处理阶段利用局部区域内其他关键点的预测来校正当前候选框的置信度。在KITTI数据集上的大量实验证明了我们SGCCNet的通用性和有效性。在KITTI test集上，SGCCNet在Moderate难度级别的AP_{3D}指标上达到了80.82%，优于所有其他基于点的检测器，分别超过IA-SSD和Fast Point R-CNN 2.35%和3.42%。此外，SGCCNet对其他基于点的检测器表现出优异的可移植性。
英文摘要：
The single-stage point-based 3D object detectors have attracted widespread research interest due to their advantages of lightweight and fast inference speed. However, they still face challenges such as inadequate learning of low-quality objects (ILQ) and misalignment between localization accuracy and classification confidence (MLC). In this paper, we propose SGCCNet to alleviate these two issues. For ILQ, SGCCNet adopts a Saliency-Guided Data Augmentation (SGDA) strategy to enhance the robustness of the model on low-quality objects by reducing its reliance on salient features. Specifically, We construct a classification task and then approximate the saliency scores of points by moving points towards the point cloud centroid in a differentiable process. During the training process, SGCCNet will be forced to learn from low saliency features through dropping points. Meanwhile, to avoid internal covariate shift and contextual features forgetting caused by dropping points, we add a geometric normalization module and skip connection block in each stage. For MLC, we design a Confidence Correction Mechanism (CCM) specifically for point-based multi-class detectors. This mechanism corrects the confidence of the current proposal by utilizing the predictions of other key points within the local region in the post-processing stage. Extensive experiments on the KITTI dataset demonstrate the generality and effectiveness of our SGCCNet. On the KITTI \textit{test} set, SGCCNet achieves $80.82\%$ for the metric of $AP_{3D}$ on the \textit{Moderate} level, outperforming all other point-based detectors, surpassing IA-SSD and Fast Point R-CNN by $2.35\%$ and $3.42\%$, respectively. Additionally, SGCCNet demonstrates excellent portability for other point-based detectors
We propose an augmentation policy for Contrastive Self-Supervised Learning (SSL) in the form of an already established Salient Image Segmentation technique entitled Global Contrast based Salient Region Detection. This detection technique, which had been devised for unrelated Computer Vision tasks, was empirically observed to play the role of an augmentation facilitator within the SSL protocol. This observation is rooted in our practical attempts to learn, by SSL-fashion, aerial imagery of solar panels, which exhibit challenging boundary patterns. Upon the successful integration of this techniq…
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation Chinese Academy of Sciences（中国科学院自动化研究所）；Huazhong University of Science and Technology（华中科技大学）；Horizon Robotics（地平线机器人）
Data augmentation is a powerful technique to increase the diversity of data, which can effectively improve the generalization ability of neural networks in image recognition tasks. Recent data mixing based augmentation strategies have achieved great success. Especially, CutMix uses a simple but effective method to improve the classifiers by randomly cropping a patch from one image and pasting it on another image. To further promote the performance of CutMix, a series of works explore to use the saliency information of the image to guide the mixing. We systematically study the importance of the…
SAMAug: Point Prompt Augmentation for Segment Anything Model
School of Computing, University of Georgia（佐治亚大学计算机学院）；School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；Department of Computer Science and Engineering, Lehigh University（里海大学计算机科学与工程系）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Department of Computer Science and Engineering, University of Texas at Arlington（德克萨斯大学阿灵顿分校计算机科学与工程系）；Department of Radiation Oncology, Mayo Clinic（梅奥诊所放射肿瘤科）；School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）
Beijing Institute of Technology（北京理工大学）；University of East Anglia（东安格利亚大学）
This paper proposes a deep learning model to efficiently detect salient regions in videos. It addresses two important issues: (1) deep video saliency model training with the absence of sufficiently large and pixel-wise annotated video data, and (2) fast video saliency training and detection. The proposed deep video saliency network consists of two modules, for capturing the spatial and temporal saliency information, respectively. The dynamic saliency model, explicitly incorporating saliency estimates from the static saliency model, directly produces spatiotemporal saliency inference without ti…
University of Liverpool（利物浦大学）；Xi’an Jiaotong-Liverpool University (XJTLU)（西交利物浦大学）；Duke Kunshan University（昆山杜克大学）
Single-source domain generalization (SDG) in medical image segmentation is a challenging yet essential task as domain shifts are quite common among clinical image datasets. Previous attempts most conduct global-only/random augmentation. Their augmented samples are usually insufficient in diversity and informativeness, thus failing to cover the possible target domain distribution. In this paper, we rethink the data augmentation strategy for SDG in medical image segmentation. Motivated by the class-level representation invariance and style mutability of medical images, we hypothesize that unseen…
Data augmentation is a commonly applied technique with two seemingly related advantages. With this method one can increase the size of the training set generating new samples and also increase the invariance of the network against the applied transformations. Unfortunately all images contain both relevant and irrelevant features for classification therefore this invariance has to be class specific. In this paper we will present a new method which uses saliency maps to restrict the invariance of neural networks to certain regions, providing higher test accuracy in classification tasks.
University of Texas Austin（德克萨斯大学奥斯汀分校）；Facebook Inc（脸书公司）
Data augmentation (DA) is an essential technique for training state-of-the-art deep learning systems. In this paper, we empirically show data augmentation might introduce noisy augmented examples and consequently hurt the performance on unaugmented data during inference. To alleviate this issue, we propose a simple yet highly effective approach, dubbed \emph{KeepAugment}, to increase augmented images fidelity. The idea is first to use the saliency map to detect important regions on the original images and then preserve these informative regions during augmentation. This information-preserving …
City University of Hong Kong（香港城市大学）；Microsoft Research Asia（微软亚洲研究院）
Recent progress in contrastive learning has revolutionized unsupervised representation learning. Concretely, multiple views (augmentations) from the same image are encouraged to map to the similar embeddings, while views from different images are pulled apart. In this paper, through visualizing and diagnosing classification errors, we observe that current contrastive models are ineffective at localizing the foreground object, limiting their ability to extract discriminative high-level features. This is due to the fact that view generation process considers pixels in an image uniformly. To addr…
Mixup is a commonly adopted data augmentation technique for image classification. Recent advances in mixup methods primarily focus on mixing based on saliency. However, many saliency detectors require intense computation and are especially burdensome for parameter-heavy transformer models. To this end, we propose TokenMixup, an efficient attention-guided token-level data augmentation method that aims to maximize the saliency of a mixed set of tokens. TokenMixup provides x15 faster saliency-aware data augmentation compared to gradient-based methods. Moreover, we introduce a variant of TokenMixu…
Tianjin University（天津大学）；Dalian University of Technology（大连理工大学）；University of South Carolina（南卡罗来纳大学）
The existing still-static deep learning based saliency researches do not consider the weighting and highlighting of extracted features from different layers, all features contribute equally to the final saliency decision-making. Such methods always evenly detect all "potentially significant regions" and unable to highlight the key salient object, resulting in detection failure of dynamic scenes. In this paper, based on the fact that salient areas in videos are relatively small and concentrated, we propose a \textbf{key salient object re-augmentation method (KSORA) using top-down semantic knowl…
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Samsung AI Centre Toronto（三星多伦多人工智能中心）；Borealis AI（Borealis人工智能公司）；York University（约克大学）
Data augmentation is a key element for training accurate models by reducing overfitting and improving generalization. For image classification, the most popular data augmentation techniques range from simple photometric and geometrical transformations, to more complex methods that use visual saliency to craft new training examples. As augmentation methods get more complex, their ability to increase the test accuracy improves, yet, such methods become cumbersome, inefficient and lead to poor out-of-domain generalization, as we show in this paper. This motivates a new augmentation technique that…
Korea University（高丽大学）；Meta Reality Labs（Meta现实实验室）
Data augmentation is key to improving the generalization ability of deep learning models. Mixup is a simple and widely-used data augmentation technique that has proven effective in alleviating the problems of overfitting and data scarcity. Also, recent studies of saliency-aware Mixup in the image domain show that preserving discriminative parts is beneficial to improving the generalization performance. However, these Mixup-based data augmentations are underexplored in 3D vision, especially in point clouds. In this paper, we propose SageMix, a saliency-guided Mixup for point clouds to preserve …
Shanghai Jiao Tong University（上海交通大学）；MarkableAI Inc（MarkableAI公司）；Baidu Research（百度研究院）；West Virginia University（西弗吉尼亚大学）；Université de Nantes（南特大学）
Data size is the bottleneck for developing deep saliency models, because collecting eye-movement data is very time consuming and expensive. Most of current studies on human attention and saliency modeling have used high quality stereotype stimuli. In real world, however, captured images undergo various types of transformations. Can we use these transformations to augment existing saliency datasets? Here, we first create a novel saliency dataset including fixations of 10 observers over 1900 images degraded by 19 types of transformations. Second, by analyzing eye movements, we find that observer…
Contrastive learning has emerged as a competitive pretraining method for object detection. Despite this progress, there has been minimal investigation into the robustness of contrastively pretrained detectors when faced with domain shifts. To address this gap, we conduct an empirical study of contrastive learning and out-of-domain object detection, studying how contrastive view design affects robustness. In particular, we perform a case study of the detection-focused pretext task Instance Localization (InsLoc) and propose strategies to augment views and enhance robustness in appearance-shifted…
EventRPG: Event Data Augmentation with Relevance Propagation Guidance
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Northeastern University（东北大学）；Seeing Machines（Seeing Machines公司）；Monash University（莫纳什大学）；Peking University（北京大学）
The University of Tokyo（东京大学）；National Institute of Informatics（国立信息学研究所）
360° images are informative -- it contains omnidirectional visual information around the camera. However, the areas that cover a 360° image is much larger than the human's field of view, therefore important information in different view directions is easily overlooked. To tackle this issue, we propose a method for predicting the optimal set of Region of Interest (RoI) from a single 360° image using the visual saliency as a clue. To deal with the scarce, strongly biased training data of existing single 360° image saliency prediction dataset, we also propose a data augmentation method based on t…
SGD-Mix: Enhancing Domain-Specific Image Classification with Label-Preserving Data Augmentation
Department of Computer Science and Information Engineering, National Cheng Kung University（计算机科学与信息工程系，国立成功大学）；Department of Biomedical Informatics, Harvard Medical School, Harvard University（生物医学信息学系，哈佛医学院，哈佛大学）
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；AITRICS
The Mixup scheme suggests mixing a pair of samples to create an augmented training sample and has gained considerable attention recently for improving the generalizability of neural networks. A straightforward and widely used extension of Mixup is to combine with regional dropout-like methods: removing random patches from a sample and replacing it with the features from another sample. Albeit their simplicity and effectiveness, these methods are prone to create harmful samples due to their randomness. To address this issue, 'maximum saliency' strategies were recently proposed: they select only…
Seoul National University（首尔国立大学）；Neural Processing Research Center（神经处理研究中心）
While deep neural networks show great performance on fitting to the training distribution, improving the networks' generalization performance to the test distribution and robustness to the sensitivity to input perturbations still remain as a challenge. Although a number of mixup based augmentation strategies have been proposed to partially address them, it remains unclear as to how to best utilize the supervisory signal within each input data for mixup from the optimization perspective. We propose a new perspective on batch mixup and formulate the optimal construction of a batch of mixup data …
Deep learning has driven remarkable accuracy increases in many computer vision problems. One ongoing challenge is how to achieve the greatest accuracy in cases where training data is limited. A second ongoing challenge is that trained models oftentimes do not generalize well even to new data that is subjectively similar to the training set. We address these challenges in a novel way, with the first-ever (to our knowledge) exploration of encoding human judgement about salient regions of images into the training data. We compare the accuracy and generalization of a state-of-the-art deep learning…
RaffeSDG: Random Frequency Filtering enabled Single-source Domain Generalization for Medical Image Segmentation
Faculty of Biomedical Engineering, Shenzhen University of Advanced Technology（深圳先进技术大学生物医学工程学院）；The Department of Computer Science and Engineering, Southern University of Science and Technology（南方科技大学计算机科学与工程系）；The Department of Biostatistics, School of Global Public Health, New York University（纽约大学全球公共卫生学院生物统计学系）；Department of Ophthalmology, Peking University People's Hospital（北京大学人民医院眼科部门）；Beijing Key Laboratory of Ocular Disease and Optometry Science, Peking University People’s Hospital（北京大学人民医院眼病与视光学科学重点实验室）
Nanjing University of Science and Technology（南京理工大学）
Unsupervised anomaly detection is a challenging task in industrial applications since it is impracticable to collect sufficient anomalous samples. In this paper, a novel Self-Supervised Guided Segmentation Framework (SGSF) is proposed by jointly exploring effective generation method of forged anomalous samples and the normal sample features as the guidance information of segmentation for anomaly detection. Specifically, to ensure that the generated forged anomaly samples are conducive to model training, the Saliency Augmentation Module (SAM) is proposed. SAM introduces a saliency map to genera…
Data-Augmentation (DA) is known to improve performance across tasks and datasets. We propose a method to theoretically analyze the effect of DA and study questions such as: how many augmented samples are needed to correctly estimate the information encoded by that DA? How does the augmentation policy impact the final parameters of a model? We derive several quantities in close-form, such as the expectation and variance of an image, loss, and model's output under a given DA distribution. Those derivations open new avenues to quantify the benefits and limitations of DA. For example, we show that…
Interpretability-guided Data Augmentation for Robust Segmentation in Multi-centre Colonoscopy Data
The Netherlands Cancer Institute（荷兰癌症研究所）；GROW School for Oncology and Developmental Biology, Maastricht University Medical Center（马斯特里赫特大学医学中心GROW肿瘤与发育生物学学院）
As 3D point cloud analysis has received increasing attention, the insufficient scale of point cloud datasets and the weak generalization ability of networks become prominent. In this paper, we propose a simple and effective augmentation method for the point cloud data, named PointCutMix, to alleviate those problems. It finds the optimal assignment between two point clouds and generates new training data by replacing the points in one sample with their optimal assigned pairs. Two replacement strategies are proposed to adapt to the accuracy or robustness requirement for different tasks, one of w…
