Do Neural Networks for Segmentation Understand Insideness?
Center for Brains, Minds and Machines, Massachusetts Institute of Technology（麻省理工学院大脑、心智与机器中心）；Max Planck Institute of Neurobiology（马克斯·普朗克神经生物学研究所）；Fujitsu Laboratories Ltd.（富士通实验室有限公司）
Kimberly Villalobos, Vilim Štih, Amineh Ahmadinejad, Shobhita Sundaram, Jamell Dozier, Andrew Francl, Frederico Azevedo, Tomotake Sasaki, Xavier Boix
The insideness problem is an aspect of image segmentation that consists of determining which pixels are inside and outside a region. Deep Neural Networks (DNNs) excel in segmentation benchmarks, but it is unclear if they have the ability to solve the insideness problem as it requires evaluating long-range spatial dependencies. In this paper, the insideness problem is analysed in isolation, without texture or semantic cues, such that other aspects of segmentation do not interfere in the analysis. We demonstrate that DNNs for segmentation with few units have sufficient complexity to solve insideness for any curve. Yet, such DNNs have severe problems with learning general solutions. Only recurrent networks trained with small images learn solutions that generalize well to almost any curve. Recurrent networks can decompose the evaluation of long-range dependencies into a sequence of local operations, and learning with small images alleviates the common difficulties of training recurrent networks with a large number of unrolling steps.
Eindhoven University of Technology（埃因霍温理工大学）；University of Gent（根特大学）；TU Delft（代尔夫特理工大学）；NTU Athens（雅典国家技术大学）；Vicomtech（Vicomtech 技术中心）
This dissertation addresses visual scene understanding and enhances segmentation performance and generalization, training efficiency of networks, and holistic understanding. First, we investigate semantic segmentation in the context of street scenes and train semantic segmentation networks on combinations of various datasets. In Chapter 2 we design a framework of hierarchical classifiers over a single convolutional backbone, and train it end-to-end on a combination of pixel-labeled datasets, improving generalizability and the number of recognizable semantic concepts. Chapter 3 focuses on enric…
1st Place Winner of the 2024 Pixel-level Video Understanding in the Wild (CVPR'24 PVUW) Challenge in Video Panoptic Segmentation and Best Long Video Consistency of Video Semantic Segmentation
Device Solutions Research America, Samsung Semiconductor, Inc.（三星半导体公司美国设备解决方案研究中心）
A recent study has shown a phenomenon called neural collapse in that the within-class means of features and the classifier weight vectors converge to the vertices of a simplex equiangular tight frame at the terminal phase of training for classification. In this paper, we explore the corresponding structures of the last-layer feature centers and classifiers in semantic segmentation. Based on our empirical and theoretical analysis, we point out that semantic segmentation naturally brings contextual correlation and imbalanced distribution among classes, which breaks the equiangular and maximally …
Understanding Image2Video Domain Shift in Food Segmentation: An Instance-level Analysis on Apples
Interdisciplinary Program in Artificial Intelligence, Seoul National University（人工智能交叉学科项目，首尔国立大学）；Department of Biological Environmental Science, College of Life Science and Biotechnology, Dongguk University（生命科学与生物技术学院生物环境科学系，东国大学）；Department of Food Science and Biotechnology, Dongguk University（食品科学与生物技术系，东国大学）
In this work, we introduce panoramic panoptic segmentation as the most holistic scene understanding both in terms of field of view and image level understanding for standard camera based input. A complete surrounding understanding provides a maximum of information to the agent, which is essential for any intelligent vehicle in order to make informed decisions in a safety-critical dynamic environment such as real-world traffic. In order to overcome the lack of annotated panoramic images, we propose a framework which allows model training on standard pinhole images and transfers the learned feat…
Vision-Aware Text Features in Referring Image Segmentation: From Object Understanding to Context Understanding
The Hong Kong University of Science and Technology（香港科技大学）；Stony Brook University（石溪大学）；University of Science, VNU-HCM（胡志明市国家大学科学大学）；Vietnam National University, Ho Chi Minh City（胡志明市国家大学）；Trinity College Dublin（都柏林大学学院）
Single Network Panoptic Segmentation for Street Scene Understanding
Eindhoven University of Technology（埃因霍温理工大学）
在这项工作中，我们提出了一个用于全景分割的单一深度神经网络，其目标是为输入图像的每个像素提供类别标签，如语义分割，以及图像中特定对象的唯一标识符，然后进行实例分割。我们的网络进行联合语义和实例分割预测，并将这些组合起来形成全景格式的输出。这有两个主要好处：首先，整个全景预测是在一次通过中进行的，减少了所需的计算时间和资源;其次，通过联合学习任务，信息在两个任务之间共享，从而提高了性能。我们的网络在两个街景数据集上进行评估：Cityscapes和Mapillary Vistas。通过利用信息交换和改进合并算法，我们提高了单个网络的性能，并在Mapillary Vistas验证的Panoptic Quality（PQ）指标上获得了23.9分，输入分辨率为640 x 900像素。在Cityscapes验证中，我们的方法在512 x 1024像素的输入分辨率下获得了45.9的PQ分数。此外，我们的方法相对于单独的网络将预测时间减少了2倍。
Fudan University（复旦大学）；Microsoft Cloud + AI（微软云与人工智能部门）；Microsoft Research Asia（微软亚洲研究院）
Exploring dense matching between the current frame and past frames for long-range context modeling, memory-based methods have demonstrated impressive results in video object segmentation (VOS) recently. Nevertheless, due to the lack of instance understanding ability, the above approaches are oftentimes brittle to large appearance variations or viewpoint changes resulted from the movement of objects and cameras. In this paper, we argue that instance understanding matters in VOS, and integrating it with memory-based matching can enjoy the synergy, which is intuitively sensible from the definitio…
Technical Report for ICRA 2026 GOOSE 2D Fine-Grained Semantic Segmentation Challenge: Exploring Query-Based Segmentation and Increased Spatial Context for Outdoor Scene Understanding
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Stellantis, Opel Automobile GmbH（斯特兰蒂斯，欧宝汽车有限公司）
At the heart of all automated driving systems is the ability to sense the surroundings, e.g., through semantic segmentation of LiDAR sequences, which experienced a remarkable progress due to the release of large datasets such as SemanticKITTI and nuScenes-LidarSeg. While most previous works focus on sparse segmentation of the LiDAR input, dense output masks provide self-driving cars with almost complete environment information. In this paper, we introduce MASS - a Multi-Attentional Semantic Segmentation model specifically built for dense top-view understanding of the driving scenes. Our framew…
Semantic segmentation aims to robustly predict coherent class labels for entire regions of an image. It is a scene understanding task that powers real-world applications (e.g., autonomous navigation). One important application, the use of imagery for automated semantic understanding of pedestrian environments, provides remote mapping of accessibility features in street environments. This application (and others like it) require detailed geometric information of geographical objects. Semantic segmentation is a prerequisite for this task since it maps contiguous regions of the same class as sing…
Eindhoven University of Technology（埃因霍温理工大学）；University of Amsterdam（阿姆斯特丹大学）
In this work, we introduce the new scene understanding task of Part-aware Panoptic Segmentation (PPS), which aims to understand a scene at multiple levels of abstraction, and unifies the tasks of scene parsing and part parsing. For this novel task, we provide consistent annotations on two commonly used datasets: Cityscapes and Pascal VOC. Moreover, we present a single metric to evaluate PPS, called Part-aware Panoptic Quality (PartPQ). For this new task, using the metric and annotations, we set multiple baselines by merging results of existing state-of-the-art methods for panoptic segmentation…
Understanding From Human Perspective: A Multi-agent System for Interactive Egocentric Medical Image Segmentation
School of Instrument Science and Engineering, Southeast University（东南大学仪器科学与工程学院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Sichuan University-Pittsburgh Institute, Sichuan University（四川大学匹兹堡学院）；Department of Biomedical Engineering, Case Western Reserve University（凯斯西储大学生物医学工程系）
Technical Report for ICRA 2025 GOOSE 2D Semantic Segmentation Challenge: Leveraging Color Shift Correction, RoPE-Swin Backbone, and Quantile-based Label Denoising Strategy for Robust Outdoor Scene Understanding
Institute of Intelligent Systems（智能系统研究所）；College of Artificial Intelligence, National Yang Ming Chiao Tung University（人工智能学院，阳明交通大学）；Institute of Data Science, National Cheng Kung University（数据科学研究所，成功大学）
University of Michigan-Dearborn（密歇根大学迪尔伯恩分校）；University of Maryland-College Park（马里兰大学帕克分校）；Valeo Vision Systems（法雷奥视觉系统公司）
Panoptic Segmentation aims to provide an understanding of background (stuff) and instances of objects (things) at a pixel level. It combines the separate tasks of semantic segmentation (pixel level classification) and instance segmentation to build a single unified scene understanding task. Typically, panoptic segmentation is derived by combining semantic and instance segmentation tasks that are learned separately or jointly (multi-task networks). In general, instance segmentation networks are built by adding a foreground mask estimation layer on top of object detectors or using instance clust…
Independently exploring unknown spaces or finding objects in an indoor environment is a daily but challenging task for visually impaired people. However, common 2D assistive systems lack depth relationships between various objects, resulting in difficulty to obtain accurate spatial layout and relative positions of objects. To tackle these issues, we propose HIDA, a lightweight assistive system based on 3D point cloud instance segmentation with a solid-state LiDAR sensor, for holistic indoor detection and avoidance. Our entire system consists of three hardware components, two interactive functi…
Efficient RGB-D Scene Understanding via Multi-task Adaptive Learning and Cross-dimensional Feature Guidance
School of Mechanical Engineering（机械工程学院）；Vehicle Measurement, Control and Safety Key Laboratory of Sichuan Province（四川省车辆测量控制与安全重点实验室）；National Key Laboratory for Novel Software Technology（新型软件技术国家实验室）；Shanghai Advanced Research Institute（上海先进研究院）；Hubei Key Laboratory of Modern Manufacturing Quality Engineering（湖北省现代制造质量工程重点实验室）
University of Southern California（南加州大学）；Army Research Laboratory（陆军研究实验室）
Our goal is to develop stable, accurate, and robust semantic scene understanding methods for wide-area scene perception and understanding, especially in challenging outdoor environments. To achieve this, we are exploring and evaluating a range of related technology and solutions, including AI-driven multimodal scene perception, fusion, processing, and understanding. This work reports our efforts on the evaluation of a state-of-the-art approach for semantic segmentation with multiple RGB and depth sensing data. We employ four large datasets composed of diverse urban and terrain scenes and desig…
Corals are the primary habitat-building life-form on reefs that support a quarter of the species in the ocean. A coral reef ecosystem usually consists of reefs, each of which is like a tall building in any city. These reef-building corals secrete hard calcareous exoskeletons that give them structural rigidity, and are also a prerequisite for our accurate 3D modeling and semantic mapping using advanced photogrammetric computer vision and machine learning. Underwater videography as a modern underwater remote sensing tool is a high-resolution coral habitat survey and mapping technique. In this pa…
Semantic segmentation is a scene understanding task at the heart of safety-critical applications where robustness to corrupted inputs is essential. Implicit Background Estimation (IBE) has demonstrated to be a promising technique to improve the robustness to out-of-distribution inputs for semantic segmentation models for little to no cost. In this paper, we provide analysis comparing the structures learned as a result of optimization objectives that use Softmax, IBE, and Sigmoid in order to improve understanding their relationship to robustness. As a result of this analysis, we propose combini…
Unsupervised Semantic Segmentation Facilitates Model Understanding
Max-Delbruck-Center（马克斯·德尔布鲁克中心）；Helmholtz Imaging（海德堡成像）；Humboldt-Universität zu Berlin（柏林洪堡大学）；Charité Universitätsmedizin（夏里特大学医学院）；University of Potsdam（波茨坦大学）
LTCI, Télécom Paris, Institut Polytechnique de Paris（巴黎综合理工学院电信学院LTCI实验室）
Understanding the scene around the ego-vehicle is key to assisted and autonomous driving. Nowadays, this is mostly conducted using cameras and laser scanners, despite their reduced performances in adverse weather conditions. Automotive radars are low-cost active sensors that measure properties of surrounding objects, including their relative speed, and have the key advantage of not being impacted by rain, snow or fog. However, they are seldom used for scene understanding due to the size and complexity of radar raw data and the lack of annotated datasets. Fortunately, recent open-sourced datase…
Alibaba Group（阿里巴巴集团）；University of Copenhagen（哥本哈根大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of Oxford（牛津大学）
Can our video understanding systems perceive objects when a heavy occlusion exists in a scene? To answer this question, we collect a large-scale dataset called OVIS for occluded video instance segmentation, that is, to simultaneously detect, segment, and track instances in occluded scenes. OVIS consists of 296k high-quality instance masks from 25 semantic categories, where object occlusions usually occur. While our human vision systems can understand those occluded instances by contextual reasoning and association, our experiments suggest that current video understanding systems cannot. On the…
AIFloodSense: A Global Aerial Imagery Dataset for Semantic Segmentation and Understanding of Flooded Environments
Department of Management Science and Technology（管理科学与技术系）；Hellenic Mediterranean University（希伯伦地中海大学）；Institute of Computer Science, FORTH（信息科学研究所，FORTH）
Technical Report for the ICRA 2026 GOOSE 2D Fine-Grained Semantic Segmentation Challenge: Pretraining-Diverse Ensemble of Foundation Vision Encoders for Robust Outdoor Scene Understanding
LION Lab, Hefei University of Technology（合肥工业大学 LION 实验室）；University of Macau（澳门大学）
