Mask Guided Gated Convolution for Amodal Content Completion
Doctoral School of Applied Informatics and Applied Mathematics, Óbuda University（奥布达大学应用信息学与应用数学博士学院）；John von Neumann Faculty of Informatics, Óbuda University（奥布达大学约翰·冯·诺依曼信息学院）
Kaziwa Saleh, Sándor Szénási, Zoltán Vámossy
备注：
6 pages, 4 figures
发表信息：
2024 IEEE 22nd Jubilee International Symposium on Intelligent Systems and Informatics (SISY)
We present a model to reconstruct partially visible objects. The model takes a mask as an input, which we call weighted mask. The mask is utilized by gated convolutions to assign more weight to the visible pixels of the occluded instance compared to the background, while ignoring the features of the invisible pixels. By drawing more attention from the visible region, our model can predict the invisible patch more effectively than the baseline models, especially in instances with uniform texture. The model is trained on COCOA dataset and two subsets of it in a self-supervised manner. The results demonstrate that our model generates higher quality and more texture-rich outputs compared to baseline models. Code is available at: https://github.com/KaziwaSaleh/mask-guided.
We consider the problem of enriching current object detection systems with veridical object sizes and relative depth estimates from a single image. There are several technical challenges to this, such as occlusions, lack of calibration data and the scale ambiguity between object size and distance. These have not been addressed in full generality in previous work. Here we propose to tackle these issues by building upon advances in object recognition and using recently created large-scale datasets. We first introduce the task of amodal bounding box completion, which aims to infer the the full ex…
Reasoning-Driven Amodal Completion: Collaborative Agents and Perceptual Evaluation
School of Computer Science and Engineering, Beihang University（计算机科学与工程学院，北京航空航天大学）；School of Software, Beihang University（软件学院，北京航空航天大学）；School of Computing and Information Systems, The University of Melbourne（计算与信息系统学院，墨尔本大学）
This paper addresses weakly supervised amodal instance segmentation, where the goal is to segment both visible and occluded (amodal) object parts, while training provides only ground-truth visible (modal) segmentations. Following prior work, we use data manipulation to generate occlusions in training images and thus train a segmenter to predict amodal segmentations of the manipulated data. The resulting predictions on training images are taken as the pseudo-ground truth for the standard training of Mask-RCNN, which we use for amodal instance segmentation of test images. For generating the pseu…
Universitat Pompeu Fabra (UPF)（庞培法布拉大学）；Centre de Visió per Computador (CVC)（计算机视觉中心）；DxO Labs（DxO实验室）
This paper presents a computational model to recover the most likely interpretation of the 3D scene structure from a planar image, where some objects may occlude others. The estimated scene interpretation is obtained by integrating some global and local cues and provides both the complete disoccluded objects that form the scene and their ordering according to depth. Our method first computes several distal scenes which are compatible with the proximal planar image. To compute these different hypothesized scenes, we propose a perceptually inspired object disocclusion method, which works by mini…
TACO: Taming Diffusion for in-the-wild Video Amodal Completion
State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）；State Key Laboratory of General Artificial Intelligence, BIGAI（BIGAI 通用人工智能国家重点实验室）；Tsinghua University（清华大学）
Occlusion-Aware Temporally Consistent Amodal Completion for 3D Human-Object Interaction Reconstruction
Elmore Family School of Electrical and Computer Engineering（电气与计算机工程学院）；Purdue University（普渡大学）；Department of Artificial Intelligence（人工智能系）；Korea University（韩国大学）；Honda Research Institute USA（本田美国研究院）；School of Mechanical Engineering（机械工程学院）
The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）
Natural scene understanding is a challenging task, particularly when encountering images of multiple objects that are partially occluded. This obstacle is given rise by varying object ordering and positioning. Existing scene understanding paradigms are able to parse only the visible parts, resulting in incomplete and unstructured scene interpretation. In this paper, we investigate the problem of scene de-occlusion, which aims to recover the underlying occlusion ordering and complete the invisible parts of occluded objects. We make the first attempt to address the problem through a novel and un…
Learnable Shape Prototypes with Occlusion-Geometry-Guided Injection for Amodal Instance Segmentation
School of Mechanical Engineering and Automation, Northeastern University（机械工程与自动化学院，东北大学）；School of Information Science and Engineering, Northeastern University（信息科学与工程学院，东北大学）
SynergyAmodal: Deocclude Anything with Text Control
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, China（多媒体可信感知与高效计算重点实验室、教育部、厦门大学）
University of Maryland, College Park（马里兰大学帕克分校）；Facebook（脸书）
Image composition aims to blend multiple objects to form a harmonized image. Existing approaches often assume precisely segmented and intact objects. Such assumptions, however, are hard to satisfy in unconstrained scenarios. We present Amodal Instance Composition for compositing imperfect -- potentially incomplete and/or coarsely segmented -- objects onto a target image. We first develop object shape prediction and content completion modules to synthesize the amodal contents. We then propose a neural composition model to blend the objects seamlessly. Our primary technical novelty lies in using…
The University of Adelaide（阿德莱德大学）；Nanjing University of Science and Technology（南京理工大学）；Northwestern Polytechnical University（西北工业大学）；ETH Zurich（苏黎世联邦理工学院）
This paper tackles the problem of data fusion in the semantic scene completion (SSC) task, which can simultaneously deal with semantic labeling and scene completion. RGB images contain texture details of the object(s) which are vital for semantic scene understanding. Meanwhile, depth images capture geometric clues of high relevance for shape completion. Using both RGB and depth images can further boost the accuracy of SSC over employing one modality in isolation. We propose a 3D gated recurrent fusion network (GRFNet), which learns to adaptively select and fuse the relevant information from de…
School of EIC, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）；ByteDance Inc（字节跳动公司）
In this paper, we tackle the problem of human de-occlusion which reasons about occluded segmentation masks and invisible appearance content of humans. In particular, a two-stage framework is proposed to estimate the invisible portions and recover the content inside. For the stage of mask completion, a stacked network structure is devised to refine inaccurate masks from a general instance segmentation model and predict integrated masks simultaneously. Additionally, the guidance from human parsing and typical pose masks are leveraged to bring prior information. For the stage of content recovery,…
Mask-adaptive Gated Convolution and Bi-directional Progressive Fusion Network for Depth Completion
College of Information Science and Engineering, Northeastern University（东北大学信息科学与工程学院）；Foshan Graduate School of Innovation, Northeastern University（东北大学佛山创新研究生院）；National Frontiers Science Center for Industrial Intelligence and Systems Optimization, Northeastern University（东北大学工业智能与系统优化国家前沿科学中心）
Look Through Masks: Towards Masked Face Recognition with De-Occlusion Distillation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；State Key Laboratory of Virtual Reality Technology and Systems, SCSE, Beihang University（北京航空航天大学虚拟现实技术与系统国家重点实验室）；Peng Cheng Laboratory（鹏城实验室）
Memory-Guided Point Cloud Completion for Dental Reconstruction
College of Information and Intelligent Science, Donghua University, Shanghai, China（信息与智能科学学院，东华大学，上海，中国）；Institute of Artificial Intelligence, Shanghai University, Shanghai, China（人工智能研究院，上海大学，上海，中国）
Depth completion aims to recover a dense depth map from the sparse depth data and the corresponding single RGB image. The observed pixels provide the significant guidance for the recovery of the unobserved pixels' depth. However, due to the sparsity of the depth data, the standard convolution operation, exploited by most of existing methods, is not effective to model the observed contexts with depth values. To address this issue, we propose to adopt the graph propagation to capture the observed spatial contexts. Specifically, we first construct multiple graphs at different scales from observed…
