In this paper, we solve the sample shortage problem in the human parsing task. We begin with the self-learning strategy, which generates pseudo-labels for unlabeled data to retrain the model. However, directly using noisy pseudo-labels will cause error amplification and accumulation. Considering the topology structure of human body, we propose a trainable graph reasoning method that establishes internal structural connections between graph nodes to correct two typical errors in the pseudo-labels, i.e., the global structural error and the local consistency error. For the global error, we first transform category-wise features into a high-level graph model with coarse-grained structural information, and then decouple the high-level graph to reconstruct the category features. The reconstructed features have a stronger ability to represent the topology structure of the human body. Enlarging the receptive field of features can effectively reducing the local error. We first project feature pixels into a local graph model to capture pixel-wise relations in a hierarchical graph manner, then reverse the relation information back to the pixels. With the global structural and local consistency modules, these errors are rectified and confident pseudo-labels are generated for retraining. Extensive experiments on the LIP and the ATR datasets demonstrate the effectiveness of our global and local rectification modules. Our method outperforms other state-of-the-art methods in supervised human parsing tasks.
Exemplar-based class-incremental learning is to recognize new classes while not forgetting old ones, whose samples can only be saved in limited memory. The ratio fluctuation of new samples to old exemplars, which is caused by the variation of memory capacity at different environments, will bring challenges to stabilize the incremental optimization process. To address this problem, we propose a novel self-paced imbalance rectification scheme, which dynamically maintains the incremental balance during the representation learning phase. Specifically, our proposed scheme consists of a frequency co…
Beihang University（北京航空航天大学）；University of Durham（杜伦大学）
Recently, fully convolutional neural networks (FCNs) have shown significant performance in image parsing, including scene parsing and object parsing. Different from generic object parsing tasks, hand parsing is more challenging due to small size, complex structure, heavy self-occlusion and ambiguous texture problems. In this paper, we propose a novel parsing framework, Multi-Scale Dual-Branch Fully Convolutional Network (MSDB-FCN), for hand parsing tasks. Our network employs a Dual-Branch architecture to extract features of hand area, paying attention on the hand itself. These features are use…
Zhejiang University（浙江大学）；Stevens Institute of Technology（史蒂文斯理工学院）；Alibaba Group（阿里巴巴集团）；Yunnan University（云南大学）
In this paper, we investigate a novel deep-model reusing task. Our goal is to train a lightweight and versatile student model, without human-labelled annotations, that amalgamates the knowledge and masters the expertise of two pretrained teacher models working on heterogeneous problems, one on scene parsing and the other on depth estimation. To this end, we propose an innovative training strategy that learns the parameters of the student intertwined with the teachers, achieved by 'projecting' its amalgamated features onto each teacher's domain and computing the loss. We also introduce two opti…
Human Object Interaction (HOI) detection is a challenging task that requires to distinguish the interaction between a human-object pair. Attention based relation parsing is a popular and effective strategy utilized in HOI. However, current methods execute relation parsing in a "bottom-up" manner. We argue that the independent use of the bottom-up parsing strategy in HOI is counter-intuitive and could lead to the diffusion of attention. Therefore, we introduce a novel knowledge-guided top-down attention into HOI, and propose to model the relation parsing as a "look and search" process: execute …
Leveraging Labelled Data Knowledge: A Cooperative Rectification Learning Network for Semi-supervised 3D Medical Image Segmentation
School of Mechanical Engineering and Automation, Northeastern University（东北大学机械工程与自动化学院）；Australian Institute for Machine Learning, University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey（萨里大学视觉、语音与信号处理中心）
Hefei University of Technology（合肥工业大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）；Shanghai AI Lab（上海人工智能实验室）
Audio-Visual Video Parsing is a task to predict the events that occur in video segments for each modality. It often performs in a weakly supervised manner, where only video event labels are provided, i.e., the modalities and the timestamps of the labels are unknown. Due to the lack of densely annotated labels, recent work attempts to leverage pseudo labels to enrich the supervision. A commonly used strategy is to generate pseudo labels by categorizing the known event labels for each modality. However, the labels are still limited to the video level, and the temporal boundaries of event timesta…
The University of Sydney（悉尼大学）；University of Pittsburgh（匹兹堡大学）
Scene parsing is an indispensable component in understanding the semantics within a scene. Traditional methods rely on handcrafted local features and probabilistic graphical models to incorporate local and global cues. Recently, methods based on fully convolutional neural networks have achieved new records on scene parsing. An important strategy common to these methods is the aggregation of hierarchical features yielded by a deep convolutional neural network. However, typical algorithms usually aggregate hierarchical convolutional features via concatenation or linear combination, which cannot …
University of Electronic Science and Technology of China（电子科技大学）
Existing methods of multiple human parsing usually adopt a two-stage strategy (typically top-down and bottom-up), which suffers from either strong dependence on prior detection or highly computational redundancy during post-grouping. In this work, we present an end-to-end multiple human parsing framework using representative parts, termed RepParser. Different from mainstream methods, RepParser solves the multiple human parsing in a new single-stage manner without resorting to person detection or post-grouping.To this end, RepParser decouples the parsing pipeline into instance-aware kernel gene…
Reinforced Label Denoising for Weakly-Supervised Audio-Visual Video Parsing
Shandong University（山东大学）；Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））；Shandong Computer Science Center (National Supercomputer Center in Jinan)（山东省计算中心（国家超级计算济南中心））
Sun Yat-Sen University（中山大学）；ByteDance（字节跳动）；Carnegie Mellon University（卡内基梅隆大学）；Peng Cheng Laboratory（鹏城实验室）
Image-based Virtual Try-ON aims to transfer an in-shop garment onto a specific person. Existing methods employ a global warping module to model the anisotropic deformation for different garment parts, which fails to preserve the semantic information of different parts when receiving challenging inputs (e.g, intricate human poses, difficult garments). Moreover, most of them directly warp the input garment to align with the boundary of the preserved region, which usually requires texture squeezing to meet the boundary shape constraint and thus leads to texture distortion. The above inferior perf…
Australian National University（澳大利亚国立大学）；NICTA（澳大利亚国家信息与通信技术研究院）；EPFL（洛桑联邦理工学院）
Scene parsing has attracted a lot of attention in computer vision. While parametric models have proven effective for this task, they cannot easily incorporate new training data. By contrast, nonparametric approaches, which bypass any learning phase and directly transfer the labels from the training data to the query images, can readily exploit new labeled samples as they become available. Unfortunately, because of the computational cost of their label transfer procedures, state-of-the-art nonparametric methods typically filter out most training images to only keep a few relevant ones to label …
Advancing Weakly-Supervised Audio-Visual Video Parsing via Segment-wise Pseudo Labeling
Hefei University of Technology（合肥工业大学）；Shanghai AI Laboratory（上海人工智能实验室）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）；Anhui Zhonghuitong Technology Co., Ltd.（安徽中汇通科技有限公司）
University of Central Florida（中佛罗里达大学）；Istanbul Technical University（伊斯坦布尔技术大学）；MEF University（MEF大学）
Person re-identification is a challenging task mainly due to factors such as background clutter, pose, illumination and camera point of view variations. These elements hinder the process of extracting robust and discriminative representations, hence preventing different identities from being successfully distinguished. To improve the representation learning, usually, local features from human body parts are extracted. However, the common practice for such a process has been based on bounding box part detection. In this paper, we propose to adopt human semantic parsing which, due to its pixel-l…
Hunan University（湖南大学）；State University of New York at Buffalo（纽约州立大学布法罗分校）
Semantic segmentation for lightweight object parsing is a very challenging task, because both accuracy and efficiency (e.g., execution speed, memory footprint or computational complexity) should all be taken into account. However, most previous works pay too much attention to one-sided perspective, either accuracy or speed, and ignore others, which poses a great limitation to actual demands of intelligent devices. To tackle this dilemma, we propose a novel lightweight architecture named Context-Integrated and Feature-Refined Network (CIFReNet). The core components of CIFReNet are the Long-skip…
Structured prototype regularization for synthetic-to-real driving scene parsing
Tongji University（同济大学）；Beijing Institute of Aerospace Control Devices（北京航天控制设备研究所）；Ryazan State Radio Engineering University（里亚赞无线电工程大学）；Manhattan University（曼哈顿大学）
ETH Zurich（苏黎世联邦理工学院）；Inception Institute of Artificial Intelligence（人工智能 inception 研究所）；SenseTime Research（商汤科技研究院）；Tianjin University（天津大学）
Human parsing is for pixel-wise human semantic understanding. As human bodies are underlying hierarchically structured, how to model human structures is the central theme in this task. Focusing on this, we seek to simultaneously exploit the representational capacity of deep graph networks and the hierarchical human structures. In particular, we provide following two contributions. First, three kinds of part relations, i.e., decomposition, composition, and dependency, are, for the first time, completely and precisely described by three distinct relation networks. This is in stark contrast to pr…
Nanjing University（南京大学）；SenseTime Research（商汤科技研究院）；CUHK - Sensetime Joint Lab（香港中文大学-商汤联合实验室）；Shanghai AI Laboratory（上海人工智能实验室）
This paper focuses on the weakly-supervised audio-visual video parsing task, which aims to recognize all events belonging to each modality and localize their temporal boundaries. This task is challenging because only overall labels indicating the video events are provided for training. However, an event might be labeled but not appear in one of the modalities, which results in a modality-specific noisy label problem. In this work, we propose a training strategy to identify and remove modality-specific noisy labels dynamically. It is motivated by two key observations: 1) networks tend to learn …
HAPNet: Toward Superior RGB-Thermal Scene Parsing via Hybrid, Asymmetric, and Progressive Heterogeneous Feature Fusion
College of Electronic and Information Engineering, Tongji University（同济大学电子与信息工程学院）；Department of Computer Science and Engineering, Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）；MSU-BIT-SMBU Joint Research Center of Applied Mathematics, Shenzhen MSU-BIT University（深圳MSU-BIT大学应用数学联合研究中心）；Qingdao Innovation and Development Base, Harbin Institute of Technology (Weihai)（哈尔滨工业大学（威海）青岛创新与发展基地）；Department of Control Science and Engineering, Harbin Institute of Technology（哈尔滨工业大学控制科学与工程系）；Harbin Institute of Technology at Weihai（哈尔滨工业大学（威海））；Suzhou Research Institute, Harbin Institute of Technology（哈尔滨工业大学苏州研究所）；Faculty of Computational Mathematics and Cybernetics, Lomonosov Moscow State University（莫斯科罗蒙诺夫莫斯科国立大学计算数学与自动化系）
Exploring the Robustness of Human Parsers Towards Common Corruptions
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Sun Yat-sen University（中山大学）；Westlake University（西湖大学）；OPPO；Tsinghua University（清华大学）
University of Technology Sydney（悉尼科技大学）；Baidu Research（百度研究院）
Labeling pixel-level masks for fine-grained semantic segmentation tasks, e.g. human parsing, remains a challenging task. The ambiguous boundary between different semantic parts and those categories with similar appearance usually are confusing, leading to unexpected noises in ground truth masks. To tackle the problem of learning with label noises, this work introduces a purification strategy, called Self-Correction for Human Parsing (SCHP), to progressively promote the reliability of the supervised labels as well as the learned models. In particular, starting from a model trained with inaccura…
Adaptive Dual-Teacher Distillation with Subnetwork Rectification for Bridging Semantic Gaps in Black-Box Domain Adaptation
School of Computer Science and Engineering, Tianjin University of Technology（天津理工大学计算机科学与工程学院）；School of Artificial Intelligence, Tianjin University（天津大学人工智能学院）
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Parsing articulated objects, e.g. humans and animals, into semantic parts (e.g. body, head and arms, etc.) from natural images is a challenging and fundamental problem for computer vision. A big difficulty is the large variability of scale and location for objects and their corresponding parts. Even limited mistakes in estimating scale and location will degrade the parsing output and cause errors in boundary details. To tackle these difficulties, we propose a "Hierarchical Auto-Zoom Net" (HAZN) for object part parsing which adapts to the local scales of objects and parts. HAZN is a sequence of…
Two-Stream Interactive Joint Learning of Scene Parsing and Geometric Vision Tasks
College of Electronic and Information Engineering, Tongji University（同济大学电子信息学院）；Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学智能自主系统研究所）；College of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；Department of Vehicle Control System and Software Development, NIO（蔚来汽车车辆控制系统与软件开发部门）；Department of Automotive Engineering, Jilin University（吉林大学汽车工程学院）；Key Laboratory of Embedded System and Service Computing (Ministry of Education), Tongji University（同济大学嵌入式系统与服务计算重点实验室）；College of Electronic and Information Engineering, Shanghai Institute of Intelligent Science and Technology（上海智能科学与技术研究院电子信息学院）；Shanghai Key Laboratory of Intelligent Autonomous Systems（上海智能自主系统重点实验室）
Télécom-Paris, Intitute Polytechnique de Paris（巴黎理工学院电信巴黎）；Shanghai Jiao Tong University（上海交通大学）；University of Trento（特伦托大学）；Fondazione Bruno Kessler（布鲁诺·凯塞勒基金会）
In this work we address multi-target domain adaptation (MTDA) in semantic segmentation, which consists in adapting a single model from an annotated source dataset to multiple unannotated target datasets that differ in their underlying data distributions. To address MTDA, we propose a self-training strategy that employs pseudo-labels to induce cooperation among multiple domain-specific classifiers. We employ feature stylization as an efficient way to generate image views that forms an integral part of self-training. Additionally, to prevent the network from overfitting to noisy pseudo-labels, w…
For a given video-based Human-Object Interaction scene, modeling the spatio-temporal relationship between humans and objects are the important cue to understand the contextual information presented in the video. With the effective spatio-temporal relationship modeling, it is possible not only to uncover contextual information in each frame but also to directly capture inter-time dependencies. It is more critical to capture the position changes of human and objects over the spatio-temporal dimension when their appearance features may not show up significant changes over time. The full use of ap…
Technical Report for ICRA 2025 GOOSE 2D Semantic Segmentation Challenge: Leveraging Color Shift Correction, RoPE-Swin Backbone, and Quantile-based Label Denoising Strategy for Robust Outdoor Scene Understanding
Institute of Intelligent Systems（智能系统研究所）；College of Artificial Intelligence, National Yang Ming Chiao Tung University（人工智能学院，阳明交通大学）；Institute of Data Science, National Cheng Kung University（数据科学研究所，成功大学）
Label-anticipated Event Disentanglement for Audio-Visual Video Parsing
Hefei University of Technology（合肥工业大学）；Anhui Zhonghuitong Technology Co., Ltd.（安徽中汇通科技有限公司）；University of Science and Technology of China（中国科学技术大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Northwestern Polytechnical University（西北工业大学）；Shanghai AI Laboratory（上海人工智能实验室）
