Context modeling is one of the most fertile subfields of visual recognition which aims at designing discriminant image representations while incorporating their intrinsic and extrinsic relationships. However, the potential of context modeling is currently underexplored and most of the existing solutions are either context-free or restricted to simple handcrafted geometric relationships. We introduce in this paper DHCN: a novel Deep Hierarchical Context Network that leverages different sources of contexts including geometric and semantic relationships. The proposed method is based on the minimization of an objective function mixing a fidelity term, a context criterion and a regularizer. The solution of this objective function defines the architecture of a bi-level hierarchical context network; the first level of this network captures scene geometry while the second one corresponds to semantic relationships. We solve this representation learning problem by training its underlying deep network whose parameters correspond to the most influencing bi-level contextual relationships and we evaluate its performances on image annotation using the challenging ImageCLEF benchmark.
This report demonstrates our solution for the Open Images 2018 Challenge. Based on our detailed analysis on the Open Images Datasets (OID), it is found that there are four typical features: large-scale, hierarchical tag system, severe annotation incompleteness and data imbalance. Considering these characteristics, an amount of strategies are employed, including SNIPER, soft sampling, class-aware sampling (CAS), hierarchical non-maximum suppression (HNMS) and so on. In virtue of these effective strategies, and further using the powerful SENet154 armed with feature pyramid module and deformable …
Many tasks such as retrieval and recommendations can significantly benefit from structuring the data, commonly in a hierarchical way. To achieve this through annotations of high dimensional data such as images or natural text can be significantly labor intensive. We propose an approach for uncovering the hierarchical structure of data based on efficient discriminative testing rather than annotations of individual datapoints. Using two-alternative-forced-choice (2AFC) testing and deep metric learning we achieve embedding of the data in semantic space where we are able to successfully hierarchic…
P. Universidad Catolica de Chile（智利天主教大学）；Stanford University（斯坦福大学）；Universidad del Norte（北方大学）
In this paper, we introduce a new hierarchical model for human action recognition using body joint locations. Our model can categorize complex actions in videos, and perform spatio-temporal annotations of the atomic actions that compose the complex action being performed.That is, for each atomic action, the model generates temporal action annotations by estimating its starting and ending times, as well as, spatial annotations by inferring the human body parts that are involved in executing the action. our model includes three key novel properties: (i) it can be trained with no spatial supervis…
One of the main obstacles to 3D semantic segmentation is the significant amount of endeavor required to generate expensive point-wise annotations for fully supervised training. To alleviate manual efforts, we propose GIDSeg, a novel approach that can simultaneously learn segmentation from sparse annotations via reasoning global-regional structures and individual-vicinal properties. GIDSeg depicts global- and individual- relation via a dynamic edge convolution network coupled with a kernelized identity descriptor. The ensemble effects are obtained by endowing a fine-grained receptive field to a…
The increase in data collection has made data annotation an interesting and valuable task in the contemporary world. This paper presents a new methodology for quickly annotating data using click-supervision and hierarchical object detection. The proposed work is semi-automatic in nature where the task of annotations is split between the human and a neural network. We show that our improved method of annotation reduces the time, cost and mental stress on a human annotator. The research also highlights how our method performs better than the current approach in different circumstances such as va…
Peng Cheng Laboratory（鹏城实验室）；Harbin Institute of Technology（哈尔滨工业大学）
This paper aims to solve the video object segmentation (VOS) task in a scribble-supervised manner, in which VOS models are not only trained by the sparse scribble annotations but also initialized with the sparse target scribbles for inference. Thus, the annotation burdens for both training and initialization can be substantially lightened. The difficulties of scribble-supervised VOS lie in two aspects. On the one hand, it requires the powerful ability to learn from the sparse scribble annotations during training. On the other hand, it demands strong reasoning capability during inference given …
TACO is an open image dataset for litter detection and segmentation, which is growing through crowdsourcing. Firstly, this paper describes this dataset and the tools developed to support it. Secondly, we report instance segmentation performance using Mask R-CNN on the current version of TACO. Despite its small size (1500 images and 4784 annotations), our results are promising on this challenging problem. However, to achieve satisfactory trash detection in the wild for deployment, TACO still needs much more manual annotations. These can be contributed using: http://tacodataset.org/
Context plays an important role in visual pattern recognition as it provides complementary clues for different learning tasks including image classification and annotation. In the particular scenario of kernel learning, the general recipe of context-based kernel design consists in learning positive semi-definite similarity functions that return high values not only when data share similar content but also similar context. However, in spite of having a positive impact on performance, the use of context in these kernel design methods has not been fully explored; indeed, context has been handcraf…
3D automatic annotation has received increased attention since manually annotating 3D point clouds is laborious. However, existing methods are usually complicated, e.g., pipelined training for 3D foreground/background segmentation, cylindrical object proposals, and point completion. Furthermore, they often overlook the inter-object feature relation that is particularly informative to hard samples for 3D annotation. To this end, we propose a simple yet effective end-to-end Context-Aware Transformer (CAT) as an automated 3D-box labeler to generate precise 3D box annotations from 2D boxes, traine…
Understanding the context of complex and cluttered scenes is a challenging problem for semantic segmentation. However, it is difficult to model the context without prior and additional supervision because the scene's factors, such as the scale, shape, and appearance of objects, vary considerably in these scenes. To solve this, we propose to learn the structures of objects and the hierarchy among objects because context is based on these intrinsic properties. In this study, we design novel hierarchical, contextual, and multiscale pyramidal representations to capture the properties from an input…
University of Utah（犹他大学）；Scientific Computing and Imaging Institute（科学计算与成像研究所）
Scene labeling is the problem of assigning an object label to each pixel. It unifies the image segmentation and object recognition problems. The importance of using contextual information in scene labeling frameworks has been widely realized in the field. We propose a contextual framework, called contextual hierarchical model (CHM), which learns contextual information in a hierarchical framework for scene labeling. At each level of the hierarchy, a classifier is trained based on downsampled input images and outputs of previous levels. Our model then incorporates the resulting multi-resolution …
Scribble-Supervised Medical Image Segmentation with Dynamic Teacher Switching and Hierarchical Consistency
1 Carnegie Mellon University, PA 15213, United States 2 PASSIO Lab, North Carolina A\&T State University, NC 27401, United States 3 University of Houston, TX 77004, United States 4 Northwestern University, IL 60611, United States
CoTZero: Annotation-Free Human-Like Vision Reasoning via Hierarchical Synthetic CoT
University of Electronic Science and Technology of China（电子科技大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong MMLab（香港中文大学 MMLab）；The College of Computer Science and Technology（计算机科学与技术学院）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
Learning-based Hierarchical Tracheal Anatomy Understanding from Sparse Surgical Demonstration Annotations for Ultrasound Robots
The Chinese University of Hong Kong（香港中文大学）；Imperial College London（伦敦帝国学院）；Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院）；SLAI Shenzhen Loop Area Institute（深圳环域先进人工智能研究院）
Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局资讯通信研究院）；Nanyang Technological University（南洋理工大学）
Deep learning has achieved promising segmentation performance on 3D left atrium MR images. However, annotations for segmentation tasks are expensive, costly and difficult to obtain. In this paper, we introduce a novel hierarchical consistency regularized mean teacher framework for 3D left atrium segmentation. In each iteration, the student model is optimized by multi-scale deep supervision and hierarchical consistency regularization, concurrently. Extensive experiments have shown that our method achieves competitive performance as compared with full annotation, outperforming other state-of-the…
Deep EM with Hierarchical Latent Label Modelling for Multi-Site Prostate Lesion Segmentation
UCL Hawkes Institute（UCL霍克斯研究所）；Department of Medical Physics and Biomedical Engineering（医学物理与生物医学工程系）；University College London（伦敦大学学院）；Centre for Medical Imaging（医学成像中心）；Division of Medicine（医学系）；Division of Surgery and Interventional Science（外科与介入科学系）；Cancer Institute（癌症研究所）；Urology Department（泌尿科部门）；UCLH（UCLH医院）；Radiology Department（放射科部门）；Imperial College Healthcare（帝国学院医疗）
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；Megvii Research Nanjing, Megvii Technology（旷视科技南京研究院）
State-of-the-art two-stage object detectors apply a classifier to a sparse set of object proposals, relying on region-wise features extracted by RoIPool or RoIAlign as inputs. The region-wise features, in spite of aligning well with the proposal locations, may still lack the crucial context information which is necessary for filtering out noisy background detections, as well as recognizing objects possessing no distinctive appearances. To address this issue, we present a simple but effective Hierarchical Context Embedding (HCE) framework, which can be applied as a plug-and-play component, to f…
Skip and Skip: Segmenting Medical Images with Prompts
Academy for Engineering and Technology, Fudan University（复旦大学工程与技术学院）；Engineering Research Center of AI and Robotics, Ministry of Education, China（教育部人工智能与机器人工程研究中心）；Jilin Provincial Key Laboratory of Intelligence Science（吉林省智能科学重点实验室）
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote SensingWuhan University（武汉大学测绘遥感信息工程国家重点实验室）
Global context information is vital in visual understanding problems, especially in pixel-level semantic segmentation. The mainstream methods adopt the self-attention mechanism to model global context information. However, pixels belonging to different classes usually have weak feature correlation. Modeling the global pixel-level correlation matrix indiscriminately is extremely redundant in the self-attention mechanism. In order to solve the above problem, we propose a hierarchical context network to differentially model homogeneous pixels with strong correlations and heterogeneous pixels with…
Clinical Center（临床中心）；National Library of Medicine（美国国家医学图书馆）；National Institutes of Health（美国国立卫生研究院）
Despite the recent advances in automatically describing image contents, their applications have been mostly limited to image caption datasets containing natural images (e.g., Flickr 30k, MSCOCO). In this paper, we present a deep learning model to efficiently detect a disease from an image and annotate its contexts (e.g., location, severity and the affected organs). We employ a publicly available radiology dataset of chest x-rays and their reports, and use its image annotations to mine disease names to train convolutional neural networks (CNNs). In doing so, we adopt various regularization tech…
GenCAMO: Scene-Graph Contextual Decoupling for Environment-aware and Mask-free Camouflage Image-Dense Annotation Generation
China University of Petroleum (East China)（中国石油大学（华东））；Southwest Jiaotong University（西南交通大学）；Beijing Information Science and Technology University（北京信息科技大学）；Beihang University（北航）
Existing semantic segmentation models heavily rely on dense pixel-wise annotations. To reduce the annotation pressure, we focus on a challenging task named zero-shot semantic segmentation, which aims to segment unseen objects with zero annotations. This task can be accomplished by transferring knowledge across categories via semantic word embeddings. In this paper, we propose a novel context-aware feature generation method for zero-shot segmentation named CaGNet. In particular, with the observation that a pixel-wise feature highly depends on its contextual information, we insert a contextual m…
MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations
Shanghai AI Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；Zhejiang University（浙江大学）；The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；Zhiyuan College, Shanghai Jiao Tong University（上海交通大学紫阳学院）；CPII under InnoHK（创新香港下的CPII）
We consider the problem of how to improve automatic target recognition by fusing the naive sensor-level classification decisions with "intuition," or context, in a mathematically principled way. This is a general approach that is compatible with many definitions of context, but for specificity, we consider context as co-occurrence in imagery. In particular, we consider images that contain multiple objects identified at various confidence levels. We learn the patterns of co-occurrence in each context, then use these patterns as hyper-parameters for a Hierarchical Bayesian Model. The result is t…
Large collections of geo-referenced panoramic images are freely available for cities across the globe, as well as detailed maps with location and meta-data on a great variety of urban objects. They provide a potentially rich source of information on urban objects, but manual annotation for object detection is costly, laborious and difficult. Can we utilize such multimedia sources to automatically annotate street level images as an inexpensive alternative to manual labeling? With the PanorAMS framework we introduce a method to automatically generate bounding box annotations for panoramic images…
Xiamen University（厦门大学）；Chinese PLA General Hospital（中国人民解放军总医院）；Deepwise AI Laboratory（深睿智慧人工智能实验室）
The analysis of organ vessels is essential for computer-aided diagnosis and surgical planning. But it is not a easy task since the fine-detailed connected regions of organ vessel bring a lot of ambiguity in vessel segmentation and sub-type recognition, especially for the low-contrast capillary regions. Furthermore, recent two-staged approaches would accumulate and even amplify these inaccuracies from the first-stage whole vessel segmentation into the second-stage sub-type vessel pixel-wise classification. Moreover, the scarcity of manual annotation in organ vessels poses another challenge. In …
Federal University of Technology - Parana（巴拉那联邦理工大学）；University of Maryland, College Park（马里兰大学帕克分校）
Reasoning about images/objects and their hierarchical interactions is a key concept for the next generation of computer vision approaches. Here we present a new framework to deal with it through a visual hierarchical context-based reasoning. Current reasoning methods use the fine-grained labels from images' objects and their interactions to predict labels to new objects. Our framework modifies this current information flow. It goes beyond and is independent of the fine-grained labels from the objects to define the image context. It takes into account the hierarchical interactions between diffe…
From Histopathology Images to Cell Clouds: Learning Slide Representations with Hierarchical Cell Transformer
University of Science and Technology Beijing（北京科技大学）；DAMO Academy, Alibaba Group（阿里巴巴达摩院）；Zhejiang University（浙江大学）；Fudan University（复旦大学）；Changhai Hospital（长海医院）；Peking University（北京大学）
This paper proposes a new methodology to automatically build semantic hierarchies suitable for image annotation and classification. The building of the hierarchy is based on a new measure of semantic similarity. The proposed measure incorporates several sources of information: visual, conceptual and contextual as we defined in this paper. The aim is to provide a measure that best represents image semantics. We then propose rules based on this measure, for the building of the final hierarchy, and which explicitly encode hierarchical relationships between different concepts. Therefore, the built…
Instituto Politécnico Nacional（国立理工学院）；National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）
Multimodal sentiment analysis is a very actively growing field of research. A promising area of opportunity in this field is to improve the multimodal fusion mechanism. We present a novel feature fusion strategy that proceeds in a hierarchical fashion, first fusing the modalities two in two and only then fusing all three modalities. On multimodal sentiment analysis of individual utterances, our strategy outperforms conventional concatenation of features by 1%, which amounts to 5% reduction in error rate. On utterance-level multimodal sentiment analysis of multi-utterance video clips, for which…
LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation
AIML, Adelaide University（AIML，阿德莱德大学）；East China Normal University（华东师范大学）；NERC-RVC, Hunan University（NERC-RVC，湖南大学）；The University of Western Australia（西澳大学）；Breaker Industries
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Wireless tags are increasingly used to track and identify common items of interest such as retail goods, food, medicine, clothing, books, documents, keys, equipment, and more. At the same time, there is a need for labelled visual data featuring such items for the purpose of training object detection and recognition models for robots operating in homes, warehouses, stores, libraries, pharmacies, and so on. In this paper, we ask: can we leverage the tracking and identification capabilities of such tags as a basis for a large-scale automatic image annotation system for robotic perception tasks? W…
