Hierarchical segmentation entails creating segmentations at varying levels of granularity. We introduce the first hierarchical semantic segmentation dataset with subpart annotations for natural images, which we call SPIN (SubPartImageNet). We also introduce two novel evaluation metrics to evaluate how well algorithms capture spatial and semantic relationships across hierarchical levels. We benchmark modern models across three different tasks and analyze their strengths and weaknesses across objects, parts, and subparts. To facilitate community-wide progress, we publicly release our dataset at https://joshmyersdean.github.io/spin/index.html.
Socialized UAV Cross-Task Learning: Towards Cross-Granularity Collaboration through Hierarchical Interaction
Kunming University of Science and Technology（昆明理工大学）；Tianjin University（天津大学）；University of New South Wales（新南威尔士大学）；Tianjin University of Sport（天津体育学院）；Southeast University（东南大学）；Hebei University of Technology（河北工业大学）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Capsule networks are designed to present the objects by a set of parts and their relationships, which provide an insight into the procedure of visual perception. Although recent works have shown the success of capsule networks on simple objects like digits, the human faces with homologous structures, which are suitable for capsules to describe, have not been explored. In this paper, we propose a Hierarchical Parsing Capsule Network (HP-Capsule) for unsupervised face subpart-part discovery. When browsing large-scale face images without labels, the network first encodes the frequently observed p…
University of Chinese Academy of Sciences（中国科学院大学）；ByteDance Inc（字节跳动公司）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Albeit with varying degrees of progress in the field of Semi-Supervised Semantic Segmentation, most of its recent successes are involved in unwieldy models and the lightweight solution is still not yet explored. We find that existing knowledge distillation techniques pay more attention to pixel-level concepts from labeled data, which fails to take more informative cues within unlabeled data into account. Consequently, we offer the first attempt to provide lightweight SSSS models via a novel multi-granularity distillation (MGD) scheme, where multi-granularity is captured from three aspects: i) …
In this paper, we introduce a novel hierarchical aggregation design that captures different levels of temporal granularity in action recognition. Our design principle is coarse-to-fine and achieved using a tree-structured network; as we traverse this network top-down, pooling operations are getting less invariant but timely more resolute and well localized. Learning the combination of operations in this network -- which best fits a given ground-truth -- is obtained by solving a constrained minimization problem whose solution corresponds to the distribution of weights that capture the contribut…
Hierarchical multi-granularity classification (HMC) assigns hierarchical multi-granularity labels to each object and focuses on encoding the label hierarchy, e.g., ["Albatross", "Laysan Albatross"] from coarse-to-fine levels. However, the definition of what is fine-grained is subjective, and the image quality may affect the identification. Thus, samples could be observed at any level of the hierarchy, e.g., ["Albatross"] or ["Albatross", "Laysan Albatross"], and examples discerned at coarse categories are often neglected in the conventional setting of HMC. In this paper, we study the HMC probl…
Unsupervised semantic segmentation aims to discover groupings within and across images that capture object and view-invariance of a category without external supervision. Grouping naturally has levels of granularity, creating ambiguity in unsupervised segmentation. Existing methods avoid this ambiguity and treat it as a factor outside modeling, whereas we embrace it and desire hierarchical grouping consistency for unsupervised segmentation. We approach unsupervised segmentation as a pixel-wise feature learning problem. Our idea is that a good representation shall reveal not just a particular l…
LUMOS: Universal Semi-Supervised OCT Retinal Layer Segmentation with Hierarchical Reliable Mutual Learning
Southern University of Science and Technology（南方科技大学）；Shenzhen Loop Area Institute（深圳河套学院）；Jiaxing Research Institute, Southern University of Science and Technology（南方科技大学嘉兴研究院）
VADTree: Explainable Training-Free Video Anomaly Detection via Hierarchical Granularity-Aware Tree
School of Software, Xi’an Jiaotong University（西安交通大学软件学院）；China Railway Xi’an Group（中国铁路西安集团）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
Harbin Engineering University（哈尔滨工程大学）；NC State University（北卡罗来纳州立大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
This paper proposes to learn hierarchical compositional AND-OR model for interpretable image synthesis by sparsifying the generator network. The proposed method adopts the scene-objects-parts-subparts-primitives hierarchy in image representation. A scene has different types (i.e., OR) each of which consists of a number of objects (i.e., AND). This can be recursively formulated across the scene-objects-parts-subparts hierarchy and is terminated at the primitive level (e.g., wavelets-like basis). To realize this AND-OR hierarchy in image synthesis, we learn a generator network that consists of t…
Adaptive Hierarchical Graph Cut for Multi-granularity Out-of-distribution Detection
Interdisciplinary Graduate Programme, Nanyang Technological University（新加坡国立大学跨学科研究生项目）；College of Computing and Data Science, Nanyang Technological University（新加坡国立大学计算与数据科学学院）；KINDI Computing Research Center, College of Engineering, Qatar University（卡塔尔大学工程学院KINDI计算研究中心）
Zhejiang University（浙江大学）；Zhejiang Gongshang University（浙江工商大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）；Huazhong University of Science and Technology（华中科技大学）；Alibaba Group（阿里巴巴集团）
This paper aims for the language-based product image retrieval task. The majority of previous works have made significant progress by designing network structure, similarity measurement, and loss function. However, they typically perform vision-text matching at certain granularity regardless of the intrinsic multiple granularities of images. In this paper, we focus on the cross-modal similarity measurement, and propose a novel Hierarchical Similarity Learning (HSL) network. HSL first learns multi-level representations of input data by stacked encoders, and object-granularity similarity and ima…
HiFi-Syn: Hierarchical Granularity Discrimination for High-Fidelity Synthesis of MR Images with Structure Preservation
Ruijin Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属瑞金医院）；College of Health Science and Technology, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院健康科学技术学院）；Clinical Neuroscience Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属瑞金医院临床神经科学中心）；National Engineering Research Center of Advanced Magnetic Resonance Technologies for Diagnosis and Therapy, Shanghai Jiao Tong University（上海交通大学先进磁共振诊断与治疗技术国家工程研究中心）；Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）
CoHD: A Counting-Aware Hierarchical Decoding Framework for Generalized Referring Expression Segmentation
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；School of EIC, Huazhong University of Science and Technology（华中科技大学电子信息与工程学院）；Tencent TEG（腾讯技术工程事业群）
University of California, San Diego（加州大学圣地亚哥分校）；Adobe Research（Adobe研究院）
Document images are a ubiquitous source of data where the text is organized in a complex hierarchical structure ranging from fine granularity (e.g., words), medium granularity (e.g., regions such as paragraphs or figures), to coarse granularity (e.g., the whole page). The spatial hierarchical relationships between content at different levels of granularity are crucial for document image understanding tasks. Existing methods learn features from either word-level or region-level but fail to consider both simultaneously. Word-level models are restricted by the fact that they originate from pure-t…
Subject-Aware Multi-Granularity Alignment for Zero-Shot EEG-to-Image Retrieval
School of Automation, Hangzhou Dianzi University（杭州电子科技大学自动化学院）；Zhejiang Provincial Key Laboratory of Brain Computer Collaborative Intelligence Technology and Applications, Hangzhou, Zhejiang（浙江省脑机协同智能技术与应用重点实验室，杭州，浙江）；College of Computer Science and Technology, Hangzhou Dianzi University（杭州电子科技大学计算机科学与技术学院）
LapFM: A Laparoscopic Segmentation Foundation Model via Hierarchical Concept Evolving Pre-training
School of Computer Science, University of Lincoln, UK（英国林肯大学计算机科学学院）；University of Nottingham, UK（英国诺丁汉大学）；University of Nottingham Ningbo China, China（中国宁波诺丁汉大学）；University of Strasbourg, France（法国斯特拉斯堡大学）；Technical University of Munich, Germany（德国慕尼黑技术大学）；Graduate School of Information, Production and Systems, Waseda University, Japan（日本早稻田大学信息、生产与系统研究生院）；Department of Gastrointestinal Surgery, The Second Qilu Hospital, Shandong University, China（中国山东大学第二齐鲁医院胃肠外科）；School of Engineering and Physical Science, University of Lincoln, Lincoln LN6 7TS, UK（英国林肯大学工程与物理科学学院）；Yale University, New Haven, CT 06510, USA（美国耶鲁大学）
ETH Zurich（苏黎世联邦理工学院）；Agroscope（瑞士农业研究中心）；Federal Office for Agriculture, Switzerland（瑞士联邦农业办公室）；University of Zurich（苏黎世大学）
The aim of this paper is to map agricultural crops by classifying satellite image time series. Domain experts in agriculture work with crop type labels that are organised in a hierarchical tree structure, where coarse classes (like orchards) are subdivided into finer ones (like apples, pears, vines, etc.). We develop a crop classification method that exploits this expert knowledge and significantly improves the mapping of rare crop types. The three-level label hierarchy is encoded in a convolutional, recurrent neural network (convRNN), such that for each pixel the model predicts three labels a…
Xidian University（西安电子科技大学）；Guangdong OPPO Mobile Telecommunications Corp., Ltd（广东欧珀移动通信有限公司）
Image-text retrieval (ITR) is a challenging task in the field of multimodal information processing due to the semantic gap between different modalities. In recent years, researchers have made great progress in exploring the accurate alignment between image and text. However, existing works mainly focus on the fine-grained alignment between image regions and sentence fragments, which ignores the guiding significance of context background information. Actually, integrating the local fine-grained information and global context background information can provide more semantic clues for retrieval. …
Advancing Medical Radiograph Representation Learning: A Hybrid Pre-training Paradigm with Multilevel Semantic Granularity
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；The University of Georgia（佐治亚大学）；Northwestern Polytechnical University（西北工业大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
University of Freiburg（弗莱堡大学）；University of Maryland Baltimore County（马里兰大学巴尔的摩县分校）
Many real-world video-text tasks involve different levels of granularity, such as frames and words, clip and sentences or videos and paragraphs, each with distinct semantics. In this paper, we propose a Cooperative hierarchical Transformer (COOT) to leverage this hierarchy information and model the interactions between different levels of granularity and different modalities. The method consists of three major components: an attention-aware feature aggregation layer, which leverages the local temporal context (intra-level, e.g., within a clip), a contextual transformer to learn the interaction…
The Chinese University of Hong Kong（香港中文大学）；SmartMore（思谋科技）；The University of Hong Kong（香港大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））
Few-shot semantic segmentation (FSS) aims to form class-agnostic models segmenting unseen classes with only a handful of annotations. Previous methods limited to the semantic feature and prototype representation suffer from coarse segmentation granularity and train-set overfitting. In this work, we design Hierarchically Decoupled Matching Network (HDMNet) mining pixel-level support correlation based on the transformer architecture. The self-attention modules are used to assist in establishing hierarchical dense features, as a means to accomplish the cascade matching between query and support f…
We introduce MeronymNet, a novel hierarchical approach for controllable, part-based generation of multi-category objects using a single unified model. We adopt a guided coarse-to-fine strategy involving semantically conditioned generation of bounding box layouts, pixel-level part layouts and ultimately, the object depictions themselves. We use Graph Convolutional Networks, Deep Recurrent Networks along with custom-designed Conditional Variational Autoencoders to enable flexible, diverse and category-aware generation of 2-D objects in a controlled manner. The performance scores for generated ob…
