Panoptic segmentation aims to perform instance segmentation for foreground instances and semantic segmentation for background stuff simultaneously. The typical top-down pipeline concentrates on two key issues: 1) how to effectively model the intrinsic interaction between semantic segmentation and instance segmentation, and 2) how to properly handle occlusion for panoptic segmentation. Intuitively, the complementarity between semantic segmentation and instance segmentation can be leveraged to improve the performance. Besides, we notice that using detection/mask scores is insufficient for resolving the occlusion problem. Motivated by these observations, we propose a novel deep panoptic segmentation scheme based on a bidirectional learning pipeline. Moreover, we introduce a plug-and-play occlusion handling algorithm to deal with the occlusion between different object instances. The experimental results on COCO panoptic benchmark validate the effectiveness of our proposed method. Codes will be released soon at https://github.com/Mooonside/BANet.
Panoptic segmentation requires segments of both "things" (countable object instances) and "stuff" (uncountable and amorphous regions) within a single output. A common approach involves the fusion of instance segmentation (for "things") and semantic segmentation (for "stuff") into a non-overlapping placement of segments, and resolves overlaps. However, instance ordering with detection confidence do not correlate well with natural occlusion relationship. To resolve this issue, we propose a branch that is tasked with modeling how two instance masks should overlap one another as a binary relation.…
Panoptic segmentation presents a new challenge in exploiting the merits of both detection and segmentation, with the aim of unifying instance segmentation and semantic segmentation in a single framework. However, an efficient solution for panoptic segmentation in the emerging domain of LiDAR point cloud is still an open research problem and is very much under-explored. In this paper, we present a fast and robust LiDAR point cloud panoptic segmentation framework, referred to as Panoptic-PolarNet. We learn both semantic segmentation and class-agnostic instance clustering in a single inference ne…
Zhejiang University（浙江大学）；Megvii Inc. (Face++)（旷视科技有限公司（Face++））；Huazhong University of Science and Technology（华中科技大学）；Peking University（北京大学）；The University of Tokyo（东京大学）
Panoptic segmentation, which needs to assign a category label to each pixel and segment each object instance simultaneously, is a challenging topic. Traditionally, the existing approaches utilize two independent models without sharing features, which makes the pipeline inefficient to implement. In addition, a heuristic method is usually employed to merge the results. However, the overlapping relationship between object instances is difficult to determine without sufficient context information during the merging process. To address the problems, we propose a novel end-to-end network for panopti…
Amodal panoptic segmentation aims to connect the perception of the world to its cognitive understanding. It entails simultaneously predicting the semantic labels of visible scene regions and the entire shape of traffic participant instances, including regions that may be occluded. In this work, we formulate a proposal-free framework that tackles this task as a multi-label and multi-class problem by first assigning the amodal masks to different layers according to their relative occlusion order and then employing amodal instance regression on each layer independently while learning background s…
The behavioural research of pigs can be greatly simplified if automatic recognition systems are used. Especially systems based on computer vision have the advantage that they allow an evaluation without affecting the normal behaviour of the animals. In recent years, methods based on deep learning have been introduced and have shown pleasingly good results. Especially object and keypoint detectors have been used to detect the individual animals. Despite good results, bounding boxes and sparse keypoints do not trace the contours of the animals, resulting in a lot of information being lost. There…
SED-MVS: Segmentation-Driven and Edge-Aligned Deformation Multi-View Stereo with Depth Restoration and Occlusion Constraint
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；School of Electrical Engineering and Computer Science, The University of Queensland（昆士兰大学电气工程与计算机科学学院）；Hunan Police Academy（湖南警察学院）；Cooperative MediaNet Innovation Center, Shanghai Jiao Tong University（上海交通大学协同媒体网络创新中心）；School of Information Science & Engineering, Lanzhou University（兰州大学信息科学与工程学院）；Agricultural Information Institute, Chinese Academy of Agricultural Sciences（中国农业科学院农业信息研究所）
National Taiwan University（国立台湾大学）；National University of Kaohsiung（国立高雄大学）
Recently, there has been a panoptic segmentation task combining semantic and instance segmentation, in which the goal is to classify each pixel with the corresponding instance ID. In this work, we propose a solution to tackle the panoptic segmentation task. The overall structure combines the bottom-up method and the top-down method. Therefore, not only can there be better performance, but also the execution speed can be maintained. The network mainly pays attention to the quality of the mask. In the previous work, we can see that the uneven contour of the object is more likely to appear, resul…
University of Freiburg（弗莱堡大学）；Toyota Research Institute（丰田研究院）
Panoptic segmentation of point clouds is a crucial task that enables autonomous vehicles to comprehend their vicinity using their highly accurate and reliable LiDAR sensors. Existing top-down approaches tackle this problem by either combining independent task-specific networks or translating methods from the image domain ignoring the intricacies of LiDAR data and thus often resulting in sub-optimal performance. In this paper, we present the novel top-down Efficient LiDAR Panoptic Segmentation (EfficientLPS) architecture that addresses multiple challenges in segmenting LiDAR point clouds includ…
We present Adaptive Instance Selection network architecture for class-agnostic instance segmentation. Given an input image and a point $(x, y)$, it generates a mask for the object located at $(x, y)$. The network adapts to the input point with a help of AdaIN layers, thus producing different masks for different objects on the same image. AdaptIS generates pixel-accurate object masks, therefore it accurately segments objects of complex shape or severely occluded ones. AdaptIS can be easily combined with standard semantic segmentation pipeline to perform panoptic segmentation. To illustrate the …
Southeast University（东南大学）；Microsoft Research Asia（微软亚洲研究院）；The Johns Hopkins University（约翰斯·霍普金斯大学）
Occlusion is probably the biggest challenge for human pose estimation in the wild. Typical solutions often rely on intrusive sensors such as IMUs to detect occluded joints. To make the task truly unconstrained, we present AdaFuse, an adaptive multiview fusion method, which can enhance the features in occluded views by leveraging those in visible views. The core of AdaFuse is to determine the point-point correspondence between two views which we solve effectively by exploring the sparsity of the heatmap representation. We also learn an adaptive fusion weight for each camera view to reflect its …
This paper presents a unified framework to (i) locate the ball, (ii) predict the pose, and (iii) segment the instance mask of players in team sports scenes. Those problems are of high interest in automated sports analytics, production, and broadcast. A common practice is to individually solve each problem by exploiting universal state-of-the-art models, \eg, Panoptic-DeepLab for player segmentation. In addition to the increased complexity resulting from the multiplication of single-task models, the use of the off-the-shelf models also impedes the performance due to the complexity and specifici…
Efficient RGB-D Scene Understanding via Multi-task Adaptive Learning and Cross-dimensional Feature Guidance
School of Mechanical Engineering（机械工程学院）；Vehicle Measurement, Control and Safety Key Laboratory of Sichuan Province（四川省车辆测量控制与安全重点实验室）；National Key Laboratory for Novel Software Technology（新型软件技术国家实验室）；Shanghai Advanced Research Institute（上海先进研究院）；Hubei Key Laboratory of Modern Manufacturing Quality Engineering（湖北省现代制造质量工程重点实验室）
A Comprehensive Survey on Video Scene Parsing:Advances, Challenges, and Prospects
College of Software, Nankai University Tianjin China；School of Electrical；Electronic Engineering, NTU \& Institute for Infocomm Research, A STAR Singapore；School of Information；College of Computer Science, Nankai University Tianjin China；VCIP, College of Computer Science, Nankai University Tianjin China；Academy for Advanced Interdisciplinary Studies, Nankai University Tianjin China；Nankai International Advanced Research Institute Shenzhen Futian China；College of Software, Nankai University；Electronic Engineering, NTU \& Institute for Infocomm Research, A STAR；College of Computer Science, Nankai University；VCIP, College of Computer Science, Nankai University；Academy for Advanced Interdisciplinary Studies, Nankai University；Nankai International Advanced Research Institute
