Over-sampling De-occlusion Attention Network for Prohibited Items Detection in Noisy X-ray Images
Beihang University（北京航空航天大学）；Shen Yuan Honors College, Beihang University（北航沈元荣誉学院）；Beijing Advanced Innovation Center for Big Data-Based Precision Medicine（北京大数据精准医学高精尖创新中心）
Security inspection is X-ray scanning for personal belongings in suitcases, which is significantly important for the public security but highly time-consuming for human inspectors. Fortunately, deep learning has greatly promoted the development of computer vision, offering a possible way of automatic security inspection. However, items within a luggage are randomly overlapped resulting in noisy X-ray images with heavy occlusions. Thus, traditional CNN-based models trained through common image recognition datasets fail to achieve satisfactory performance in this scenario. To address these problems, we contribute the first high-quality prohibited X-ray object detection dataset named OPIXray, which contains 8885 X-ray images from 5 categories of the widely-occurred prohibited item ``cutters''. The images are gathered from an airport and these prohibited items are annotated manually by professional inspectors, which can be used as a benchmark for model training and further facilitate future research. To better improve occluded X-ray object detection, we further propose an over-sampling de-occlusion attention network (DOAM-O), which consists of a novel de-occlusion attention module and a new over-sampling training strategy. Specifically, our de-occlusion module, namely DOAM, simultaneously leverages the different appearance information of the prohibited items; the over-sampling training strategy forces the model to put more emphasis on these hard samples consisting these items of high occlusion levels, which is more suitable for this scenario. We comprehensively evaluated DOAM-O on the OPIXray dataset, which proves that our model can stably improve the performance of the famous detection models such as SSD, YOLOv3, and FCOS, and outperform many extensively-used attention mechanisms.
Prohibited Items Segmentation via Occlusion-aware Bilayer Modeling
Department of Computing, The Hong Kong Polytechnic University（计算系，香港理工大学）；Research Institute of Multiple Agents and Embodied Intelligence（多智能体与具身智能研究院）；Pengcheng Laboratory（鹏城实验室）
Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；University of North Texas（北德克萨斯大学）
Automatic security inspection relying on computer vision technology is a challenging task in real-world scenarios due to many factors, such as intra-class variance, class imbalance, and occlusion. Most previous methods rarely touch the cases where the prohibited items are deliberately hidden in messy objects because of the scarcity of large-scale datasets, hindering their applications. To address this issue and facilitate related research, we present a large-scale dataset, named PIDray, which covers various cases in real-world scenarios for prohibited item detection, especially for deliberatel…
X-ray Computed Tomography (CT) based 3D imaging is widely used in airports for aviation security screening whilst prior work on prohibited item detection focuses primarily on 2D X-ray imagery. In this paper, we aim to evaluate the possibility of extending the automatic prohibited item detection from 2D X-ray imagery to volumetric 3D CT baggage security screening imagery. To these ends, we take advantage of 3D Convolutional Neural Neworks (CNN) and popular object detection frameworks such as RetinaNet and Faster R-CNN in our work. As the first attempt to use 3D CNN for volumetric 3D CT baggage …
AO-DETR: Anti-Overlapping DETR for X-Ray Prohibited Items Detection
College of Information Science and Engineering, Northeastern University（东北大学信息科学与工程学院）；Foshan Graduate School of Innovation, Northeastern University（东北大学佛山创新研究生院）
Detecting prohibited items in X-ray security imagery is pivotal in maintaining border and transport security against a wide range of threat profiles. Convolutional Neural Networks (CNN) with the support of a significant volume of data have brought advancement in such automated prohibited object detection and classification. However, collating such large volumes of X-ray security imagery remains a significant challenge. This work opens up the possibility of using synthetically composed imagery, avoiding the need to collate such large volumes of hand-annotated real-world imagery. Here we investi…
We proposed a new modeling method to promote the performance of prohibited items recognition via X-ray image. We analyzed the characteristics of prohibited items and X-ray images. We found the fact that the scales of some items are too small to be recognized which encumber the model performance. Then we adopted a set of data augmentation and modified the model to adapt the field of prohibited items recognition. The Convolutional Block Attention Module(CBAM) and rescoring mechanism has been assembled into the model. By the modification, our model achieved a mAP of 89.9% on SIXray10, mAP of 74.8…
State Key Lab of Software Development Environment, Beihang University（北京航空航天大学软件开发环境国家重点实验室）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, Beihang University（北京航空航天大学大数据精准医疗北京高精尖创新中心）
Security inspection often deals with a piece of baggage or suitcase where objects are heavily overlapped with each other, resulting in an unsatisfactory performance for prohibited items detection in X-ray images. In the literature, there have been rare studies and datasets touching this important topic. In this work, we contribute the first high-quality object detection dataset for security inspection, named Occluded Prohibited Items X-ray (OPIXray) image benchmark. OPIXray focused on the widely-occurred prohibited item "cutter", annotated manually by professional inspectors from the internati…
State Key Laboratory of Computer Science, ISCAS（中国科学院软件研究所计算机科学国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Hangzhou Institute for Advanced Study, UCAS（中国科学院大学杭州高等研究院）；JD Finance America Corporation（京东金融美国公司）；Beihang University（北京航空航天大学）
Automatic security inspection using computer vision technology is a challenging task in real-world scenarios due to various factors, including intra-class variance, class imbalance, and occlusion. Most of the previous methods rarely solve the cases that the prohibited items are deliberately hidden in messy objects due to the lack of large-scale datasets, restricted their applications in real-world scenarios. Towards real-world prohibited item detection, we collect a large-scale dataset, named as PIDray, which covers various cases in real-world scenarios for prohibited item detection, especiall…
State Key Laboratory of Software Development Environment, Beihang University（北京航空航天大学软件开发环境国家重点实验室）；iFLYTEK Research（科大讯飞研究院）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）
Prohibited items detection in X-ray images often plays an important role in protecting public safety, which often deals with color-monotonous and luster-insufficient objects, resulting in unsatisfactory performance. Till now, there have been rare studies touching this topic due to the lack of specialized high-quality datasets. In this work, we first present a High-quality X-ray (HiXray) security inspection image dataset, which contains 102,928 common prohibited items of 8 categories. It is the largest dataset of high quality for prohibited items detection, gathered from the real-world airport …
Automatic detection of prohibited objects within passenger baggage is important for aviation security. X-ray Computed Tomography (CT) based 3D imaging is widely used in airports for aviation security screening whilst prior work on automatic prohibited item detection focus primarily on 2D X-ray imagery. These works have proven the possibility of extending deep convolutional neural networks (CNN) based automatic prohibited item detection from 2D X-ray imagery to volumetric 3D CT baggage security screening imagery. However, previous work on 3D object detection in baggage security screening imager…
The recent advancement in deep Convolutional Neural Network (CNN) has brought insight into the automation of X-ray security screening for aviation security and beyond. Here, we explore the viability of two recent end-to-end object detection CNN architectures, Cascade R-CNN and FreeAnchor, for prohibited item detection by balancing processing time and the impact of image data compression from an operational viewpoint. Overall, we achieve maximal detection performance using a FreeAnchor architecture with a ResNet50 backbone, obtaining mean Average Precision (mAP) of 87.7 and 85.8 for using the O…
Khalifa University（哈利法大学）；Durham University（杜伦大学）；The University of Western Australia（西澳大利亚大学）；Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Detecting baggage threats is one of the most difficult tasks, even for expert officers. Many researchers have developed computer-aided screening systems to recognize these threats from the baggage X-ray scans. However, all of these frameworks are limited in identifying the contraband items under extreme occlusion. This paper presents a novel instance segmentation framework that utilizes trainable structure tensors to highlight the contours of the occluded and cluttered contraband items (by scanning multiple predominant orientations), while simultaneously suppressing the irrelevant baggage cont…
University of Chinese Academy of Sciences（中国科学院大学）；The Johns Hopkins University（约翰斯·霍普金斯大学）；Kingsoft（金山公司）
In this paper, we present a large-scale dataset and establish a baseline for prohibited item discovery in Security Inspection X-ray images. Our dataset, named SIXray, consists of 1,059,231 X-ray images, in which 6 classes of 8,929 prohibited items are manually annotated. It raises a brand new challenge of overlapping image data, meanwhile shares the same properties with existing datasets, including complex yet meaningless contexts and class imbalance. We propose an approach named class-balanced hierarchical refinement (CHR) to deal with these difficulties. CHR assumes that each input image is …
College of Information Science and Engineering, Northeastern University, Shenyang, 110819, China（信息科学与工程学院，东北大学，沈阳，110819，中国）；State Key Laboratory of Synthetical Automation for Process Industries, Northeastern University, Shenyang, 110819, China（过程工业综合自动化国家重点实验室，东北大学，沈阳，110819，中国）；Lee Kong Chian School of Medicine, Nanyang Technological University, 639798, Singapore（李光前医学院，南洋理工大学，639798，新加坡）
Automatic detection of prohibited items within complex and cluttered X-ray security imagery is essential to maintaining transport security, where prior work on automatic prohibited item detection focus primarily on pseudo-colour (rgb}) X-ray imagery. In this work we study the impact of variant X-ray imagery, i.e., X-ray energy response (high, low}) and effective-z compared to rgb, via the use of deep Convolutional Neural Networks (CNN) for the joint object detection and segmentation task posed within X-ray baggage security screening. We evaluate state-of-the-art CNN architectures (Mask R-CNN, …
BGM: Background Mixup for X-ray Prohibited Items Detection
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学学院）；Faculty of Data Science, City University of Macau（澳门城市大学数据科学学院）；Nuctech Company Limited（核tech公司）
针对 X 射线安检检测忽视背景信息的问题，提出基于透射成像与材料伪彩色特性的 Background Mixup 背景增强，可兼容前景增强并在多个基准上稳定提升性能。
Taming Generative Synthetic Data for X-ray Prohibited Item Detection
Institute of Information Science, Beijing Jiaotong University（信息科学学院，北京交通大学）；Faculty of Data Science, City University of Macau（数据科学学院，澳门城市大学）；Nuctech Company Limited（NuTech公司）
College of Information Science and Engineering, Northeastern University（信息科学与工程学院，东北大学）；State Key Laboratory of Synthetical Automation for Process Industries, Northeastern University（过程工业综合自动化国家重点实验室，东北大学）
Beihang University（北京航空航天大学）；Zhongguancun Laboratory（中关村实验室）；Chinese Academy of Sciences（中国科学院）；University of Science and Technology of China（中国科学技术大学）；iFLYTEK（科大讯飞）；JD Explore Academy（京东探索研究院）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）
Adversarial attacks are valuable for evaluating the robustness of deep learning models. Existing attacks are primarily conducted on the visible light spectrum (e.g., pixel-wise texture perturbation). However, attacks targeting texture-free X-ray images remain underexplored, despite the widespread application of X-ray imaging in safety-critical scenarios such as the X-ray detection of prohibited items. In this paper, we take the first step toward the study of adversarial attacks targeted at X-ray prohibited item detection, and reveal the serious threats posed by such attacks in this safety-crit…
Can a Second-View Image Be a Language? Geometric and Semantic Cross-Modal Reasoning for X-ray Prohibited Item Detection
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学学院）；State Key Laboratory of Complex & Critical Software Environment, Beihang University（北航复杂与关键软件环境国家重点实验室）
Open-Vocabulary X-ray Prohibited Item Detection via Fine-tuning CLIP
College of Information Science and Engineering, Northeastern University（东北大学信息科学与工程学院）；Foshan Graduate School of Innovation, Northeastern University（东北大学佛山创新研究院）
针对 X 射线安检中新增违禁品类别识别难题，本文提出带瓶颈适配器的 OVXD 模型，通过微调 CLIP 对齐 X 射线图像与文本，并在 PIXray、PIDray 上显著提升新类别检测性能。
Indian Institute of Technology, Kanpur（印度理工学院坎普尔分校）
In this paper, we present a semi supervised deep quick learning framework for instance detection and pixel-wise semantic segmentation of images in a dense clutter of items. The framework can quickly and incrementally learn novel items in an online manner by real-time data acquisition and generating corresponding ground truths on its own. To learn various combinations of items, it can synthesize cluttered scenes, in real time. The overall approach is based on the tutor-child analogy in which a deep network (tutor) is pretrained for class-agnostic object detection which generates labeled data fo…
X-ray imagery security screening is essential to maintaining transport security against a varying profile of threat or prohibited items. Particular interest lies in the automatic detection and classification of weapons such as firearms and knives within complex and cluttered X-ray security imagery. Here, we address this problem by exploring various end-to-end object detection Convolutional Neural Network (CNN) architectures. We evaluate several leading variants spanning the Faster R-CNN, Mask R-CNN, and RetinaNet architectures to explore the transferability of such models between varying X-ray…
Khalifa University（哈利法大学）；Durham University（杜伦大学）；The University of Western Australia（西澳大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Automated systems designed for screening contraband items from the X-ray imagery are still facing difficulties with high clutter, concealment, and extreme occlusion. In this paper, we addressed this challenge using a novel multi-scale contour instance segmentation framework that effectively identifies the cluttered contraband data within the baggage X-ray scans. Unlike standard models that employ region-based or keypoint-based techniques to generate multiple boxes around objects, we propose to derive proposals according to the hierarchy of the regions defined by the contours. The proposed fram…
Tencent AI Lab（腾讯人工智能实验室）；Tsinghua University（清华大学）
Deep Convolutional Neural Networks (CNNs) have been pushing the frontier of the face recognition research in the past years. However, existing general CNN face models generalize poorly to the scenario of occlusions on variable facial areas. Inspired by the fact that a human visual system explicitly ignores occlusions and only focuses on non-occluded facial areas, we propose a mask learning strategy to find and discard the corrupted feature elements for face recognition. A mask dictionary is firstly established by exploiting the differences between the top convoluted features of occluded and oc…
Soongsil University（崇实大学）；Korea Advanced Institute of Science and Technology（韩国科学技术院）；Nestyle Inc.（Nestyle公司）
Image-based virtual try-on aims to synthesize an image of a person wearing a given clothing item. To solve the task, the existing methods warp the clothing item to fit the person's body and generate the segmentation map of the person wearing the item before fusing the item with the person. However, when the warping and the segmentation generation stages operate individually without information exchange, the misalignment between the warped clothes and the segmentation map occurs, which leads to the artifacts in the final image. The information disconnection also causes excessive warping near th…
Giga Tech Ltd.（千兆科技有限公司）；Concordia University（康考迪亚大学）；North South University（北南大学）
Multi-class product counting and recognition identifies product items from images or videos for automated retail checkout. The task is challenging due to the real-world scenario of occlusions where product items overlap, fast movement in the conveyor belt, large similarity in overall appearance of the items being scanned, novel products, and the negative impact of misidentifying items. Further, there is a domain bias between training and test sets, specifically, the provided training dataset consists of synthetic images and the test set videos consist of foreign objects such as hands and tray.…
CTVIS: Consistent Training for Online Video Instance Segmentation
Zhejiang University（浙江大学）；Zhejiang University of Technology（浙江工业大学）；Northwest A&F University（西北农林科技大学）；The University of Adelaide（阿德莱德大学）；Swansea University（斯旺西大学）
The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）
Understanding fashion images has been advanced by benchmarks with rich annotations such as DeepFashion, whose labels include clothing categories, landmarks, and consumer-commercial image pairs. However, DeepFashion has nonnegligible issues such as single clothing-item per image, sparse landmarks (4~8 only), and no per-pixel masks, making it had significant gap from real-world scenarios. We fill in the gap by presenting DeepFashion2 to address these issues. It is a versatile benchmark of four tasks including clothes detection, pose estimation, segmentation, and retrieval. It has 801K clothing i…
Image-based virtual try-on techniques have shown great promise for enhancing the user-experience and improving customer satisfaction on fashion-oriented e-commerce platforms. However, existing techniques are currently still limited in the quality of the try-on results they are able to produce from input images of diverse characteristics. In this work, we propose a Context-Driven Virtual Try-On Network (C-VTON) that addresses these limitations and convincingly transfers selected clothing items to the target subjects even under challenging pose configurations and in the presence of self-occlusio…
Beijing University of Technology（北京工业大学）；Beijing Technology & Business University（北京工商大学）；Beijing Institute of Technology（北京理工大学）；Anhui Normal University（安徽师范大学）
Existing computer vision and object detection methods strongly rely on neural networks and deep learning. This active research area is used for applications such as autonomous driving, aerial photography, protection, and monitoring. Futuristic object detection methods rely on rectangular, boundary boxes drawn over an object to accurately locate its location. The modern object recognition algorithms, however, are vulnerable to multiple factors, such as illumination, occlusion, viewing angle, or camera rotation as well as cost. Therefore, deep learning-based object recognition will significantly…
The task of human reposing involves generating a realistic image of a person standing in an arbitrary conceivable pose. There are multiple difficulties in generating perceptually accurate images, and existing methods suffer from limitations in preserving texture, maintaining pattern coherence, respecting cloth boundaries, handling occlusions, manipulating skin generation, etc. These difficulties are further exacerbated by the fact that the possible space of pose orientation for humans is large and variable, the nature of clothing items is highly non-rigid, and the diversity in body shape diffe…
