Existing computer vision systems can compete with humans in understanding the visible parts of objects, but still fall far short of humans when it comes to depicting the invisible parts of partially occluded objects. Image amodal completion aims to equip computers with human-like amodal completion functions to understand an intact object despite it being partially occluded. The main purpose of this survey is to provide an intuitive understanding of the research hotspots, key technologies and future trends in the field of image amodal completion. Firstly, we present a comprehensive review of the latest literature in this emerging field, exploring three key tasks in image amodal completion, including amodal shape completion, amodal appearance completion, and order perception. Then we examine popular datasets related to image amodal completion along with their common data collection methods and evaluation metrics. Finally, we discuss real-world applications and future research directions for image amodal completion, facilitating the reader's understanding of the challenges of existing technologies and upcoming research trends.
We consider the problem of enriching current object detection systems with veridical object sizes and relative depth estimates from a single image. There are several technical challenges to this, such as occlusions, lack of calibration data and the scale ambiguity between object size and distance. These have not been addressed in full generality in previous work. Here we propose to tackle these issues by building upon advances in object recognition and using recently created large-scale datasets. We first introduce the task of amodal bounding box completion, which aims to infer the the full ex…
Reasoning-Driven Amodal Completion: Collaborative Agents and Perceptual Evaluation
School of Computer Science and Engineering, Beihang University（计算机科学与工程学院，北京航空航天大学）；School of Software, Beihang University（软件学院，北京航空航天大学）；School of Computing and Information Systems, The University of Melbourne（计算与信息系统学院，墨尔本大学）
Mask Guided Gated Convolution for Amodal Content Completion
Doctoral School of Applied Informatics and Applied Mathematics, Óbuda University（奥布达大学应用信息学与应用数学博士学院）；John von Neumann Faculty of Informatics, Óbuda University（奥布达大学约翰·冯·诺依曼信息学院）
Occlusion-Aware Temporally Consistent Amodal Completion for 3D Human-Object Interaction Reconstruction
Elmore Family School of Electrical and Computer Engineering（电气与计算机工程学院）；Purdue University（普渡大学）；Department of Artificial Intelligence（人工智能系）；Korea University（韩国大学）；Honda Research Institute USA（本田美国研究院）；School of Mechanical Engineering（机械工程学院）
University of Maryland, College Park（马里兰大学帕克分校）；Facebook（脸书）
Image composition aims to blend multiple objects to form a harmonized image. Existing approaches often assume precisely segmented and intact objects. Such assumptions, however, are hard to satisfy in unconstrained scenarios. We present Amodal Instance Composition for compositing imperfect -- potentially incomplete and/or coarsely segmented -- objects onto a target image. We first develop object shape prediction and content completion modules to synthesize the amodal contents. We then propose a neural composition model to blend the objects seamlessly. Our primary technical novelty lies in using…
Universitat Pompeu Fabra (UPF)（庞培法布拉大学）；Centre de Visió per Computador (CVC)（计算机视觉中心）；DxO Labs（DxO实验室）
This paper presents a computational model to recover the most likely interpretation of the 3D scene structure from a planar image, where some objects may occlude others. The estimated scene interpretation is obtained by integrating some global and local cues and provides both the complete disoccluded objects that form the scene and their ordering according to depth. Our method first computes several distal scenes which are compatible with the proximal planar image. To compute these different hypothesized scenes, we propose a perceptually inspired object disocclusion method, which works by mini…
TACO: Taming Diffusion for in-the-wild Video Amodal Completion
State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）；State Key Laboratory of General Artificial Intelligence, BIGAI（BIGAI 通用人工智能国家重点实验室）；Tsinghua University（清华大学）
AmodalSVG: Amodal Image Vectorization via Semantic Layer Peeling
School of Software, Beihang University（北京航空航天大学软件学院）；Igarashi Lab, The University of Tokyo（东京大学五十岚实验室）；Bambu Lab；Department of Computer Science, The University of Hong Kong（香港大学计算机科学系）
Amodal completion is a visual task that humans perform easily but which is difficult for computer vision algorithms. The aim is to segment those object boundaries which are occluded and hence invisible. This task is particularly challenging for deep neural networks because data is difficult to obtain and annotate. Therefore, we formulate amodal segmentation as an out-of-task and out-of-distribution generalization problem. Specifically, we replace the fully connected classifier in neural networks with a Bayesian generative model of the neural network features. The model is trained from non-occl…
IIIT Hyderabad（国际信息技术研究所海得拉巴分校）；IIT Kharagpur（印度理工学院卡拉格普尔分校）；Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）
In this paper, we address the novel, highly challenging problem of estimating the layout of a complex urban driving scenario. Given a single color image captured from a driving platform, we aim to predict the bird's-eye view layout of the road and other traffic participants. The estimated layout should reason beyond what is visible in the image, and compensate for the loss of 3D information due to projection. We dub this problem amodal scene layout estimation, which involves "hallucinating" scene layout for even parts of the world that are occluded in the image. To this end, we present MonoLay…
The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）
Natural scene understanding is a challenging task, particularly when encountering images of multiple objects that are partially occluded. This obstacle is given rise by varying object ordering and positioning. Existing scene understanding paradigms are able to parse only the visible parts, resulting in incomplete and unstructured scene interpretation. In this paper, we investigate the problem of scene de-occlusion, which aims to recover the underlying occlusion ordering and complete the invisible parts of occluded objects. We make the first attempt to address the problem through a novel and un…
SynergyAmodal: Deocclude Anything with Text Control
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, China（多媒体可信感知与高效计算重点实验室、教育部、厦门大学）
Semantic Scene Completion (SSC) aims to jointly estimate the complete geometry and semantics of a scene, assuming partial sparse input. In the last years following the multiplication of large-scale 3D datasets, SSC has gained significant momentum in the research community because it holds unresolved challenges. Specifically, SSC lies in the ambiguous completion of large unobserved areas and the weak supervision signal of the ground truth. This led to a substantially increasing number of papers on the matter. This survey aims to identify, compare and analyze the techniques providing a critical …
Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）；Samsung Research（三星研究院）；South China Normal University（华南师范大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Depth completion aims at predicting dense pixel-wise depth from an extremely sparse map captured from a depth sensor, e.g., LiDARs. It plays an essential role in various applications such as autonomous driving, 3D reconstruction, augmented reality, and robot navigation. Recent successes on the task have been demonstrated and dominated by deep learning based solutions. In this article, for the first time, we provide a comprehensive literature review that helps readers better grasp the research trends and clearly understand the current advances. We investigate the related studies from the design…
Fudan University（复旦大学）；University of Science and Technology of China（中国科学技术大学）；Shanghai AI Laboratory（上海人工智能实验室）；University of Shanghai for Science and Technology（上海理工大学）
Point cloud completion is a generation and estimation issue derived from the partial point clouds, which plays a vital role in the applications in 3D computer vision. The progress of deep learning (DL) has impressively improved the capability and robustness of point cloud completion. However, the quality of completed point clouds is still needed to be further enhanced to meet the practical utilization. Therefore, this work aims to conduct a comprehensive survey on various methods, including point-based, convolution-based, graph-based, and generative model-based approaches, etc. And this survey…
Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University（哥伦比亚大学 Mortimer B. Zuckerman 心智行为脑研究所）；Columbia University（哥伦比亚大学）
Human visual perception carves a scene at its physical joints, decomposing the world into objects, which are selectively attended, tracked, and predicted as we engage our surroundings. Object representations emancipate perception from the sensory input, enabling us to keep in mind that which is out of sight and to use perceptual content as a basis for action and symbolic cognition. Human behavioral studies have documented how object representations emerge through grouping, amodal completion, proto-objects, and object files. Deep neural network (DNN) models of visual object recognition, by cont…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Oregon（俄勒冈大学）
The development of practical applications, such as autonomous driving and robotics, has brought increasing attention to 3D point cloud understanding. While deep learning has achieved remarkable success on image-based tasks, there are many unique challenges faced by deep neural networks in processing massive, unstructured and noisy 3D points. To demonstrate the latest progress of deep learning for 3D point cloud understanding, this paper summarizes recent remarkable research contributions in this area from several different directions (classification, segmentation, detection, tracking, flow est…
Deep Learning for 3D Point Cloud Enhancement: A Survey
School of Electronics and Control Engineering, Chang’an University（长安大学电子与控制工程学院）；School of Software Engineering, Northwestern Polytechnical University（西北工业大学软件学院）；School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）
