Video segmentation for the human head and shoulders is essential in creating elegant media for videoconferencing and virtual reality applications. The main challenge is to process high-quality background subtraction in a real-time manner and address the segmentation issues under motion blurs, e.g., shaking the head or waving hands during conference video. To overcome the motion blur problem in video segmentation, we propose a novel flow-based encoder-decoder network (FUNet) that combines both traditional Horn-Schunck optical-flow estimation technique and convolutional neural networks to perform robust real-time video segmentation. We also introduce a video and image segmentation dataset: ConferenceVideoSegmentationDataset. Code and pre-trained models are available on our GitHub repository: \url{https://github.com/kuangzijian/Flow-Based-Video-Matting}.
Nagoya University（名古屋大学）；Ajou University School of Medicine（亚洲大学医学院）
This paper presents an end-to-end pixelwise fully automated segmentation of the head sectioned images of the Visible Korean Human (VKH) project based on Deep Convolutional Neural Networks (DCNNs). By converting classification networks into Fully Convolutional Networks (FCNs), a coarse prediction map, with smaller size than the original input image, can be created for segmentation purposes. To refine this map and to obtain a dense pixel-wise output, standard FCNs use deconvolution layers to upsample the coarse map. However, upsampling based on deconvolution increases the number of network param…
Tsinghua-Berkeley Shenzhen Institute, Tsinghua University（清华大学深圳国际研究生院）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Deep neural networks (DNNs) have witnessed great successes in semantic segmentation, which requires a large number of labeled data for training. We present a novel learning framework called Uncertainty guided Cross-head Co-training (UCC) for semi-supervised semantic segmentation. Our framework introduces weak and strong augmentations within a shared encoder to achieve co-training, which naturally combines the benefits of consistency and self-training. Every segmentation head interacts with its peers and, the weak augmentation result is used for supervising the strong. The consistency training …
Instance segmentation models today are very accurate when trained on large annotated datasets, but collecting mask annotations at scale is prohibitively expensive. We address the partially supervised instance segmentation problem in which one can train on (significantly cheaper) bounding boxes for all categories but use masks only for a subset of categories. In this work, we focus on a popular family of models which apply differentiable cropping to a feature map and predict a mask based on the resulting crop. Under this family, we study Mask R-CNN and discover that instead of its default strat…
Uber ATG（优步先进技术集团）；University of Toronto（多伦多大学）；The Chinese University of Hong Kong（香港中文大学）
In this paper, we propose a unified panoptic segmentation network (UPSNet) for tackling the newly proposed panoptic segmentation task. On top of a single backbone residual network, we first design a deformable convolution based semantic segmentation head and a Mask R-CNN style instance segmentation head which solve these two subtasks simultaneously. More importantly, we introduce a parameter-free panoptic head which solves the panoptic segmentation via pixel-wise classification. It first leverages the logits from the previous two heads and then innovatively expands the representation for enabl…
Instance segmentation methods require large datasets with expensive and thus limited instance-level mask labels. Partially supervised instance segmentation aims to improve mask prediction with limited mask labels by utilizing the more abundant weak box labels. In this work, we show that a class agnostic mask head, commonly used in partially supervised instance segmentation, has difficulties learning a general concept of foreground for the weakly annotated classes using box supervision only. To resolve this problem we introduce an object mask prior (OMP) that provides the mask head with the gen…
National Technical University of Ukraine “Igor Sikorsky Kyiv Polytechnic Institute”（乌克兰国立技术大学“伊戈尔·西科斯基基辅理工学院”）
This paper presents a new approach for relatively accurate brain region of interest (ROI) detection from dynamic susceptibility contrast (DSC) perfusion magnetic resonance (MR) images of a human head with abnormal brain anatomy. Such images produce problems for automatic brain segmentation algorithms, and as a result, poor perfusion ROI detection affects both quantitative measurements and visual assessment of perfusion data. In the proposed approach image segmentation is based on CUSUM filter usage that was adapted to be applicable to process DSC perfusion MR images. The result of segmentation…
Shanghai Key Laboratory of Intelligent Information Processing（上海市智能信息处理重点实验室）；School of Computer Science, Fudan University（复旦大学计算机科学学院）
Text segmentation is a challenging vision task with many downstream applications. Current text segmentation methods require pixel-level annotations, which are expensive in the cost of human labor and limited in application scenarios. In this paper, we take the first attempt to perform weakly-supervised text instance segmentation by bridging text recognition and text segmentation. The insight is that text recognition methods provide precise attention position of each text instance, and the attention location can feed to both a text adaptive refinement head (TAR) and a text segmentation head. Sp…
Weakly Supervised Intracranial Hemorrhage Segmentation using Head-Wise Gradient-Infused Self-Attention Maps from a Swin Transformer in Categorical Learning
Head and neck tumor segmentation challenge (HECKTOR) 2022 offers a platform for researchers to compare their solutions to segmentation of tumors and lymph nodes from 3D CT and PET images. In this work, we describe our solution to HECKTOR 2022 segmentation task. We re-sample all images to a common resolution, crop around head and neck region, and train SegResNet semantic segmentation network from MONAI. We use 5-fold cross validation to select best model checkpoints. The final submission is an ensemble of 15 models from 3 runs. Our solution (team name NVAUTO) achieves the 1st place on the HECKT…
Alibaba Group（阿里巴巴集团）；Amazon Prime Video（亚马逊Prime Video）
Existing semantic segmentation works have been mainly focused on designing effective decoders; however, the computational load introduced by the overall structure has long been ignored, which hinders their applications on resource-constrained hardwares. In this paper, we propose a head-free lightweight architecture specifically for semantic segmentation, named Adaptive Frequency Transformer. It adopts a parallel architecture to leverage prototype representations as specific learnable local descriptions which replaces the decoder and preserves the rich image semantics on high-resolution feature…
IMT Atlantique（大西洋国立高等矿业电信学院）；Inserm（法国国家健康与医学研究院）；University Hospital of Brest（布雷斯特大学医院）；Fondation ILDYS（ILDYS基金会）；NIH（美国国立卫生研究院）
Automatic segmentation of pathological shoulder muscles in patients with musculo-skeletal diseases is a challenging task due to the huge variability in muscle shape, size, location, texture and injury. A reliable fully-automated segmentation method from magnetic resonance images could greatly help clinicians to plan therapeutic interventions and predict interventional outcomes while eliminating time consuming manual segmentation efforts. The purpose of this work is three-fold. First, we investigate the feasibility of pathological shoulder muscle segmentation using deep learning techniques, giv…
Transmission electron microscopy (TEM) is one of the primary tools to show microstructural characterization of materials as well as film thickness. However, manual determination of film thickness from TEM images is time-consuming as well as subjective, especially when the films in question are very thin and the need for measurement precision is very high. Such is the case for head overcoat (HOC) thickness measurements in the magnetic hard disk drive industry. It is therefore necessary to develop software to automatically measure HOC thickness. In this paper, for the first time, we propose a HO…
Feature pyramid architecture has been broadly adopted in object detection and segmentation to deal with multi-scale problem. However, in this paper we show that the capacity of the architecture has not been fully explored due to the inadequate utilization of the supervision information. Such insufficient utilization is caused by the supervision signal degradation in back propagation. Thus inspired, we propose a dually supervised method, named dually supervised FPN (DSFPN), to enhance the supervision signal when training the feature pyramid network (FPN). In particular, DSFPN is constructed by …
1st Place Winner of the 2024 Pixel-level Video Understanding in the Wild (CVPR'24 PVUW) Challenge in Video Panoptic Segmentation and Best Long Video Consistency of Video Semantic Segmentation
Device Solutions Research America, Samsung Semiconductor, Inc.（三星半导体公司美国设备解决方案研究中心）
Toward Content-based Indexing and Retrieval of Head and Neck CT with Abscess Segmentation
University of Science, VNU-HCM（越南国家大学）；Vietnam National University（越南国家大学）；Thong Nhat Hospital（同奈医院）；University of Social Sciences and Humanities, VNU-HCM（越南国家人文社会大学）；Dublin City University（都柏林城市大学）；University of Dayton（戴维森大学）；John von Neumann Institute（约翰·冯·诺伊曼研究所）；University of Health Sciences, VNU-HCM（越南国家卫生科学大学）
Pubic Symphysis-Fetal Head Segmentation Network Using BiFormer Attention Mechanism and Multipath Dilated Convolution
College of Computer Science and Engineering, Chongqing University of Technology（重庆理工大学计算机科学与工程学院）；College of Artificial Intelligence, Chongqing University of Technology（重庆理工大学人工智能学院）
Few-shot segmentation (FSS) aims at performing semantic segmentation on novel classes given a few annotated support samples. With a rethink of recent advances, we find that the current FSS framework has deviated far from the supervised segmentation framework: Given the deep features, FSS methods typically use an intricate decoder to perform sophisticated pixel-wise matching, while the supervised segmentation methods use a simple linear classification head. Due to the intricacy of the decoder and its matching pipeline, it is not easy to follow such an FSS framework. This paper revives the strai…
SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Images
School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件工程学院）；School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；School of Information and Communications Engineering, Xi’an Jiaotong University（西安交通大学信息与通信工程学院）；Ministry of Education Key Lab of Intelligent Network Security, Xi’an, China（教育部智能网络安全重点实验室）；Pazhou Laboratory (Huangpu), Radarweg 29, Guangzhou, China（琶洲实验室（黄埔），雷达路29号，广州，中国）
SwinCross: Cross-modal Swin Transformer for Head-and-Neck Tumor Segmentation in PET/CT Images
Philips Research North America（飞利浦北美研究院）；Massachusetts General Hospital/Harvard Medical School（麻省总医院/哈佛医学院）；The Center for Advanced Medical Computing and Analysis, Massachusetts General Hospital/Harvard Medical School（麻省总医院/哈佛医学院高级医学计算与分析中心）
School of intelligent systems engineering, Sun Yat-sen University（中山大学智能系统工程学院）；Affiliated Hangzhou First People’s Hospital, Zhejiang University School of Medicine（浙江大学医学院附属杭州市第一人民医院）；School of Engineering, Case Western Reserve University（凯斯西储大学工程学院）
In this technical report, we present key details of our winning panoptic segmentation architecture EffPS_b1bs4_RVC. Our network is a lightweight version of our state-of-the-art EfficientPS architecture that consists of our proposed shared backbone with a modified EfficientNet-B5 model as the encoder, followed by the 2-way FPN to learn semantically rich multi-scale features. It consists of two task-specific heads, a modified Mask R-CNN instance head and our novel semantic segmentation head that processes features of different scales with specialized modules for coherent feature refinement. Fina…
A service robot serving safely and politely needs to track the surrounding people robustly, especially for Tour-Guide Robot (TGR). However, existing multi-object tracking (MOT) or multi-person tracking (MPT) methods are not applicable to TGR for the following reasons: 1. lacking relevant large-scale datasets; 2. lacking applicable metrics to evaluate trackers. In this work, we target the visual perceptual tasks for TGR and present the TGRDB dataset, a novel large-scale multi-person tracking dataset containing roughly 5.6 hours of annotated videos and over 450 long-term trajectories. Besides, w…
Accurate segmentation of prostate and surrounding organs at risk is important for prostate cancer radiotherapy treatment planning. We present a fully automated workflow for male pelvic CT image segmentation using deep learning. The architecture consists of a 2D localization network followed by a 3D segmentation network for volumetric segmentation of prostate, bladder, rectum, and femoral heads. We used a multi-channel 2D U-Net followed by a 3D U-Net with encoding arm modified with aggregated residual networks, known as ResNeXt. The models were trained and tested on a pelvic CT image dataset co…
University of Chinese Academy of Sciences（中国科学院大学）；AI Research of JD.com（京东人工智能研究院）
Person re-identification (Re-ID) aims at retrieving an input person image from a set of images captured by multiple cameras. Although recent Re-ID methods have made great success, most of them extract features in terms of the attributes of clothing (e.g., color, texture). However, it is common for people to wear black clothes or be captured by surveillance systems in low light illumination, in which cases the attributes of the clothing are severely missing. We call this problem the Black Re-ID problem. To solve this problem, rather than relying on the clothing information, we propose to exploi…
Automatic intracranial hemorrhage segmentation in 3D non-contrast head CT (NCCT) scans is significant in clinical practice. Existing hemorrhage segmentation methods usually ignores the anisotropic nature of the NCCT, and are evaluated on different in-house datasets with distinct metrics, making it highly challenging to improve segmentation performance and perform objective comparisons among different methods. The INSTANCE 2022 was a grand challenge held in conjunction with the 2022 International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI). It is intended t…
Pakistan Institute of Engineering & Applied Sciences（巴基斯坦工程与应用科学研究所）；Catholic University of Daegu（大邱天主教大学）；Air University（航空大学）；Korea University College of Medicine（高丽大学医学院）；Korea University Anam Hospital（高丽大学安岩医院）
Automatic segmentation of shoulder muscle MRI is challenging due to the high variation in muscle size, shape, texture, and spatial position of tears. Manual segmentation of tear and muscle portion is hard, time-consuming, and subjective to pathological expertise. This work proposes a new Region and Edge-based Deep Auto-Encoder (RE-DAE) for shoulder muscle MRI segmentation. The proposed RE-DAE harmoniously employs average and max-pooling operation in the encoder and decoder blocks of the Convolutional Neural Network (CNN). Region-based segmentation incorporated in the Deep Auto-Encoder (DAE) en…
Emory University（埃默里大学）；Winship Cancer Institute（温希普癌症研究所）
This paper presents a review of deep learning (DL) in multi-organ segmentation. We summarized the latest DL-based methods for medical image segmentation and applications. These methods were classified into six categories according to their network design. For each category, we listed the surveyed works, highlighted important contributions and identified specific challenges. Following the detailed review of each category, we briefly discussed its achievements, shortcomings and future potentials. We provided a comprehensive comparison among DL-based methods for thoracic and head & neck multiorga…
The University of Adelaide（阿德莱德大学）；Meituan Inc.（美团公司）；Zhejiang University（浙江大学）
We propose a simple yet effective framework for instance and panoptic segmentation, termed CondInst (conditional convolutions for instance and panoptic segmentation). In the literature, top-performing instance segmentation methods typically follow the paradigm of Mask R-CNN and rely on ROI operations (typically ROIAlign) to attend to each instance. In contrast, we propose to attend to the instances with dynamic conditional convolutions. Instead of using instance-wise ROIs as inputs to the instance mask head of fixed weights, we design dynamic instance-aware mask heads, conditioned on the insta…
Intrapartum Ultrasound Image Segmentation of Pubic Symphysis and Fetal Head Using Dual Student-Teacher Framework with CNN-ViT Collaborative Learning
College of Information Science and Technology, Jinan University（暨南大学信息科学技术学院）；Guangdong Provincial Key Laboratory of Traditional Chinese Medicine Information Technology, Jinan University（暨南大学广东省中医信息技术重点实验室）；Auckland Bioengineering Institute, University of Auckland（奥克兰大学生物工程研究所）；Universitat de Barcelona（巴塞罗那大学）；Institució Catalana de Recerca i Estudis Avançats (ICREA)（加泰罗尼亚研究与高级研究学院（ICREA））
University of Queensland（昆士兰大学）；Australian e-Health Research Centre, CSIRO（澳大利亚联邦科学与工业研究组织澳大利亚电子健康研究中心）
Data scarcity is common in deep learning models for medical image segmentation. Previous works proposed multi-dataset learning, either simultaneously or via transfer learning to expand training sets. However, medical image datasets have diverse-sized images and features, and developing a model simultaneously for multiple datasets is challenging. This work proposes Fabric Image Representation Encoding Network (FIRENet), a universal architecture for simultaneous multi-dataset segmentation and transfer learning involving arbitrary numbers of dataset(s). To handle different-sized image and feature…
Understanding the scene in which an autonomous robot operates is critical for its competent functioning. Such scene comprehension necessitates recognizing instances of traffic participants along with general scene semantics which can be effectively addressed by the panoptic segmentation task. In this paper, we introduce the Efficient Panoptic Segmentation (EfficientPS) architecture that consists of a shared backbone which efficiently encodes and fuses semantically rich multi-scale features. We incorporate a new semantic head that aggregates fine and contextual features coherently and a new var…
Isfahan University of Technology（伊斯法罕理工大学）；University of Michigan（密歇根大学）
Ultrasound imaging is a standard examination during pregnancy that can be used for measuring specific biometric parameters towards prenatal diagnosis and estimating gestational age. Fetal head circumference (HC) is one of the significant factors to determine the fetus growth and health. In this paper, a multi-task deep convolutional neural network is proposed for automatic segmentation and estimation of HC ellipse by minimizing a compound cost function composed of segmentation dice score and MSE of ellipse parameters. Experimental results on fetus ultrasound dataset in different trimesters of …
IMT Atlantique（大西洋国立高等矿业电信学校）；LaTIM UMR 1101, Inserm（法国国家健康与医学研究院LaTIM UMR 1101实验室）；Centre Hospitalier Régional et Universitaire (CHRU) de Brest（布雷斯特大学区域中心医院）；Fondation ILDYS（ILDYS基金会）；Symbiosis Center for Medical Image Analysis, Symbiosis International University（共生国际大学共生医学图像分析中心）
Clinical diagnosis of the pediatric musculoskeletal system relies on the analysis of medical imaging examinations. In the medical image processing pipeline, semantic segmentation using deep learning algorithms enables an automatic generation of patient-specific three-dimensional anatomical models which are crucial for morphological evaluation. However, the scarcity of pediatric imaging resources may result in reduced accuracy and generalization performance of individual deep segmentation models. In this study, we propose to design a novel multi-task, multi-domain learning framework in which a …
Phenotyping is the process of measuring an organism's observable traits. Manual phenotyping of crops is a labor-intensive, time-consuming, costly, and error prone process. Accurate, automated, high-throughput phenotyping can relieve a huge burden in the crop breeding pipeline. In this paper, we propose a scalable, high-throughput approach to automatically count and segment panicles (heads), a key phenotype, from aerial sorghum crop imagery. Our counting approach uses the image density map obtained from dot or region annotation as the target with a novel deep convolutional neural network archit…
Shape based classification is one of the most challenging tasks in the field of computer vision. Shapes play a vital role in object recognition. The basic shapes in an image can occur in varying scale, position and orientation. And specially when detecting human, the task becomes more challenging owing to the largely varying size, shape, posture and clothing of human. So, in our work we detect human, based on the head-shoulder shape as it is the most unvarying part of human body. Here, firstly a new and a novel equation named as the Omega Equation that describes the shape of human head-shoulde…
Rutgers University（罗格斯大学）；Institute for Hand & Arm Surgery（手与手臂外科研究所）；Rutgers-New Jersey Medical School（罗格斯大学新泽西医学院）；Sichuan University West China Hospital（四川大学华西医院）；Sichuan University（四川大学）
For diagnosis of shoulder illness, it is essential to look at the morphology deviation of scapula and humerus from the medical images that are acquired from Magnetic Resonance (MR) imaging. However, taking high-resolution MR images is time-consuming and costly because the reduction of the physical distance between image slices causes prolonged scanning time. Moreover, due to the lack of training images, images from various sources must be utilized, which creates the issue of high variance across the dataset. Also, there are human errors among the images due to the fact that it is hard to take …
ODFormer: Semantic Fundus Image Segmentation Using Transformer for Optic Nerve Head Detection
College of Electronics & Information Engineering, Tongji University（同济大学电子与信息工程学院）；Shanghai Institute of Intelligent Science and Technology, Tongji University（同济大学上海智能科学与技术研究院）；School of Life Science, Shanghai University（上海大学生命科学学院）；Tongji Hospital, School of Medicine, Tongji University（同济大学医学院附属同济医院）
Driven by the significant advancements in technology and social issues such as security management, there is a strong need for Smart Surveillance System in our society today. One of the key features of a Smart Surveillance System is efficient human detection and counting such that the system can decide and label events on its own. In this paper we propose a new, novel and robust model, The Omega Model, for detecting and counting human beings present in the scene. The proposed model employs a set of four distinct descriptors for identifying the unique features of the head, neck and shoulder reg…
CISCA and CytoDArk0: a Cell Instance Segmentation and Classification method for histo(patho)logical image Analyses and a new, open, Nissl-stained dataset for brain cytoarchitecture studies
London South Bank University（伦敦南岸大学）；University of Padova（帕多瓦大学）
