Existing face parsing methods usually misclassify occlusions as facial components. This is because occlusion is a high-level concept, it does not refer to a concrete category of object. Thus, constructing a real-world face dataset covering all categories of occlusion object is almost impossible and accurate mask annotation is labor-intensive. To deal with the problems, we present S$^3$POT, a contrast-driven framework synergizing face generation with self-supervised spatial prompting, to achieve occlusion segmentation. The framework is inspired by the insights: 1) Modern face generators' ability to realistically reconstruct occluded regions, creating an image that preserve facial geometry while eliminating occlusion, and 2) Foundation segmentation models' (e.g., SAM) capacity to extract precise mask when provided with appropriate prompts. In particular, S$^3$POT consists of three modules: Reference Generation (RF), Feature enhancement (FE), and Prompt Selection (PS). First, a reference image is produced by RF using structural guidance from parsed mask. Second, FE performs contrast of tokens between raw and reference images to obtain an initial prompt, then modifies image features with the prompt by cross-attention. Third, based on the enhanced features, PS constructs a set of positive and negative prompts and screens them with a self-attention network for a mask decoder. The network is learned under the guidance of three novel and complementary objective functions without occlusion ground truth mask involved. Extensive experiments on a dedicatedly collected dataset demonstrate S$^3$POT's superior performance and the effectiveness of each module.
SAM 3D Animal: Promptable Animal 3D Reconstruction from Images in the Wild
University of Cambridge（剑桥大学）；Southern University of Science and Technology（南方科技大学）；Tsinghua University（清华大学）；IMATI-CNR, Milan, Italy（意大利米兰IMATI-CNR研究所）
本文提出SAM 3D Animal，首个可提示的多动物3D重建框架，基于SMAL+参数化动物模型，通过关键点和掩码提示实现拥挤和遮挡场景的可靠重建，利用Herd3D数据集在多个数据集上取得最优结果。
Learning Occlusion-Robust Vision Transformers for Real-Time UAV Tracking
College of Computer Science and Engineering, Guilin University of Technology（桂林理工大学计算机科学与工程学院）；School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；School of Artificial Intelligence, Sun Yat-sen University（中山大学人工智能学院）
State Key Lab of Software Development Environment, Beihang University（北京航空航天大学软件开发环境国家重点实验室）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, Beihang University（北京航空航天大学大数据精准医疗北京高精尖创新中心）
Security inspection often deals with a piece of baggage or suitcase where objects are heavily overlapped with each other, resulting in an unsatisfactory performance for prohibited items detection in X-ray images. In the literature, there have been rare studies and datasets touching this important topic. In this work, we contribute the first high-quality object detection dataset for security inspection, named Occluded Prohibited Items X-ray (OPIXray) image benchmark. OPIXray focused on the widely-occurred prohibited item "cutter", annotated manually by professional inspectors from the internati…
The Chinese University of Hong Kong（香港中文大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Reconstructing the scene of robotic surgery from the stereo endoscopic video is an important and promising topic in surgical data science, which potentially supports many applications such as surgical visual perception, robotic surgery education and intra-operative context awareness. However, current methods are mostly restricted to reconstructing static anatomy assuming no tissue deformation, tool occlusion and de-occlusion, and camera movement. However, these assumptions are not always satisfied in minimal invasive robotic surgeries. In this work, we present an efficient reconstruction pipel…
Position: Towards Implicit Prompt For Text-To-Image Models
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；Osaka University（大阪大学）；The University of Hong Kong（香港大学）；Research Institute of Tsinghua University in Shenzhen（清华大学深圳研究院）
Improving Visual Object Tracking through Visual Prompting
Department of Computer Science, National Yang Ming Chiao Tung University（国立阳明交通大学计算机科学系）；Research Center for Information Technology Innovation, Academia Sinica, Taiwan（学术院资讯科技创新研究中心）；Microsoft, Seattle, Washington, United States（微软公司）
Doctoral School of Applied Informatics and Applied Mathematics, Obuda University, Budapest, Hungary（应用信息学与应用数学博士学院，奥布达大学，布达佩斯，匈牙利）；John von Neumann Faculty of Informatics, Obuda University, Budapest, Hungary（约翰·冯·诺依曼信息学学院，奥布达大学，布达佩斯，匈牙利）；Faculty of Economics and Informatics, J. Selye University, Komárno, Slovakia（经济与信息学学院，J. 赛利大学，科马诺，斯洛伐克）
We present a learning approach for localization and segmentation of objects in an image in a manner that is robust to partial occlusion. Our algorithm produces a bounding box around the full extent of the object and labels pixels in the interior that belong to the object. Like existing segmentation aware detection approaches, we learn an appearance model of the object and consider regions that do not fit this model as potential occlusions. However, in addition to the established use of pairwise potentials for encouraging local consistency, we use higher order potentials which capture informati…
The limited capacity to recognize faces under occlusions is a long-standing problem that presents a unique challenge for face recognition systems and even for humans. The problem regarding occlusion is less covered by research when compared to other challenges such as pose variation, different expressions, etc. Nevertheless, occluded face recognition is imperative to exploit the full potential of face recognition for real-world applications. In this paper, we restrict the scope to occluded face recognition. First, we explore what the occlusion problem is and what inherent difficulties can aris…
University of Michigan（密歇根大学）；Google Brain（谷歌大脑）；Adobe Research（奥多比研究院）
There have been remarkable successes in computer vision with deep learning. While such breakthroughs show robust performance, there have still been many challenges in learning in-depth knowledge, like occlusion or predicting physical interactions. Although some recent works show the potential of 3D data in serving such context, it is unclear how we efficiently provide 3D input to the 2D models due to the misalignment in dimensionality between 2D and 3D. To leverage the successes of 2D models in predicting self-occlusions, we design Ray-marching in Camera Space (RiCS), a new method to represent…
SED-MVS: Segmentation-Driven and Edge-Aligned Deformation Multi-View Stereo with Depth Restoration and Occlusion Constraint
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；School of Electrical Engineering and Computer Science, The University of Queensland（昆士兰大学电气工程与计算机科学学院）；Hunan Police Academy（湖南警察学院）；Cooperative MediaNet Innovation Center, Shanghai Jiao Tong University（上海交通大学协同媒体网络创新中心）；School of Information Science & Engineering, Lanzhou University（兰州大学信息科学与工程学院）；Agricultural Information Institute, Chinese Academy of Agricultural Sciences（中国农业科学院农业信息研究所）
JD Explore Academy（京东探索研究院）；Tencent Data Platform（腾讯数据平台）
With the recent advancement of deep convolutional neural networks, significant progress has been made in general face recognition. However, the state-of-the-art general face recognition models do not generalize well to occluded face images, which are exactly the common cases in real-world scenarios. The potential reasons are the absences of large-scale occluded face data for training and specific designs for tackling corrupted features brought by occlusions. This paper presents a novel face recognition method that is robust to occlusions based on a single end-to-end deep neural network. Our ap…
Depth estimation is a fundamental issue in 4-D light field processing and analysis. Although recent supervised learning-based light field depth estimation methods have significantly improved the accuracy and efficiency of traditional optimization-based ones, these methods rely on the training over light field data with ground-truth depth maps which are challenging to obtain or even unavailable for real-world light field data. Besides, due to the inevitable gap (or domain difference) between real-world and synthetic data, they may suffer from serious performance degradation when generalizing th…
SAM-pose2seg: Pose-Guided Human Instance Segmentation in Crowds
Visual Recognition Group（视觉识别组）；Department of Cybernetics（cybernetics系）；Faculty of Electrical Engineering（电气工程系）；Czech Technical University in Prague（布拉格捷克技术大学）
Australian National University（澳大利亚国立大学）；Australian Centre for Robotic Vision（澳大利亚机器人视觉中心）
Incorporating multiple camera views for detection alleviates the impact of occlusions in crowded scenes. In a multiview system, we need to answer two important questions when dealing with ambiguities that arise from occlusions. First, how should we aggregate cues from the multiple views? Second, how should we aggregate unreliable 2D and 3D spatial information that has been tainted by occlusions? To address these questions, we propose a novel multiview detection system, MVDet. For multiview aggregation, existing methods combine anchor box features from the image plane, which potentially limits …
Prompt-Driven Lightweight Foundation Model for Instance Segmentation-Based Fault Detection in Freight Trains
School of Mechanical Engineering, Hubei University of Technology（湖北工业大学机械工程学院）；Hubei Key Laboratory of Modern Manufacturing Quality Engineering, Hubei University of Technology（湖北工业大学现代制造质量工程重点实验室）；School of Mechanical Science and Engineering, Huazhong University of Science and Technology（华中科技大学机械科学与工程学院）
