We have seen significant leapfrog advancement in machine learning in recent decades. The central idea of machine learnability lies on constructing learning algorithms that learn from good data. The availability of more data being made publicly available also accelerates the growth of AI in recent years. In the domain of computer vision, the quality of image data arises from the accuracy of image annotation. Labeling large volume of image data is a daunting and tedious task. This work presents an End-to-End pipeline tool for object annotation and recognition aims at enabling quick image labeling. We have developed a modular image annotation platform which seamlessly incorporates assisted image annotation (annotation assistance), active learning and model training and evaluation. Our approach provides a number of advantages over current image annotation tools. Firstly, the annotation assistance utilizes reference hierarchy and reference images to locate the objects in the images, thus reducing the need for annotating the whole object. Secondly, images can be annotated using polygon points allowing for objects of any shape to be annotated. Thirdly, it is also interoperable across several image models, and the tool provides an interface for object model training and evaluation across a series of pre-trained models. We have tested the model and embeds several benchmarking deep learning models. The highest accuracy achieved is 74%.
Computer Vision Centre, Universitat Autonoma de Barcelona（巴塞罗那自治大学计算机视觉中心）
The ICDAR Robust Reading Competition (RRC), initiated in 2003 and re-established in 2011, has become a de-facto evaluation standard for robust reading systems and algorithms. Concurrent with its second incarnation in 2011, a continuous effort started to develop an on-line framework to facilitate the hosting and management of competitions. This paper outlines the Robust Reading Competition Annotation and Evaluation Platform, the backbone of the competitions. The RRC Annotation and Evaluation Platform is a modular framework, fully accessible through on-line interfaces. It comprises a collection …
University of Notre Dame（圣母大学）；Harper Cancer Research Institute（哈珀癌症研究所）
Image segmentation is a fundamental problem in biomedical image analysis. Recent advances in deep learning have achieved promising results on many biomedical image segmentation benchmarks. However, due to large variations in biomedical images (different modalities, image settings, objects, noise, etc), to utilize deep learning on a new application, it usually needs a new set of training data. This can incur a great deal of annotation effort and cost, because only biomedical experts can annotate effectively, and often there are too many instances in images (e.g., cells) to annotate. In this pap…
Pose estimation of the human body and hands is a fundamental problem in computer vision, and learning-based solutions require a large amount of annotated data. In this work, we improve the efficiency of the data annotation process for 3D pose estimation problems with Active Learning (AL) in a multi-view setting. AL selects examples with the highest value to annotate under limited annotation budgets (time and cost), but choosing the selection strategy is often nontrivial. We present a framework to efficiently extend existing single-view AL strategies. We then propose two novel AL strategies tha…
In recent years, supervised learning has become the dominant paradigm for training deep-learning based methods for 3D object detection. Lately, the academic community has studied 3D object detection in the context of autonomous vehicles (AVs) using publicly available datasets such as nuScenes and Argoverse 2.0. However, these datasets may have incomplete annotations, often only labeling a small subset of objects in a scene. Although commercial services exists for 3D bounding box annotation, these are often prohibitively expensive. To address these limitations, we propose ReBound, an open-sourc…
Annotation-Efficient Polyp Segmentation via Active Learning
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；The Sixth Affiliated Hospital, Sun Yat-sen University（中山大学附属第六医院）；Research Institute, Sun Yat-sen University（中山大学研究院）
Aiir Innovations（Aiir创新公司）；Delft University of Technology（代尔夫特理工大学）
This paper is on active learning where the goal is to reduce the data annotation burden by interacting with a (human) oracle during training. Standard active learning methods ask the oracle to annotate data samples. Instead, we take a profoundly different approach: we ask for annotations of the decision boundary. We achieve this using a deep generative model to create novel instances along a 1d line. A point on the decision boundary is revealed where the instances change class. Experimentally we show on three data sets that our method can be plugged-in to other active learning schemes, that hu…
When we can not assume a large amount of annotated data , active learning is a good strategy. It consists in learning a model on a small amount of annotated data (annotation budget) and in choosing the best set of points to annotate in order to improve the previous model and gain in generalization. In deep learning, active learning is usually implemented as an iterative process in which successive deep models are updated via fine tuning, but it still poses some issues. First, the initial batch of annotated images has to be sufficiently large to train a deep model. Such an assumption is strong,…
University of California, San Diego（加州大学圣迭戈分校）；Tsinghua University（清华大学）；Alibaba DAMO Academy（阿里巴巴达摩院）
Traditional depth sensors generate accurate real world depth estimates that surpass even the most advanced learning approaches trained only on simulation domains. Since ground truth depth is readily available in the simulation domain but quite difficult to obtain in the real domain, we propose a method that leverages the best of both worlds. In this paper we present a new framework, ActiveZero, which is a mixed domain learning solution for active stereovision systems that requires no real world depth annotation. First, we demonstrate the transferability of our method to out-of-distribution rea…
ESA: Annotation-Efficient Active Learning for Semantic Segmentation
The University of Adelaide（阿德莱德大学）；The Australian National University（澳大利亚国立大学）；Monash University（墨尔本大学）；La Trobe University（拉特罗布大学）；Tianjin University of Technology（天津理工大学）
Multi-task learning is central to many real-world applications. Unfortunately, obtaining labelled data for all tasks is time-consuming, challenging, and expensive. Active Learning (AL) can be used to reduce this burden. Existing techniques typically involve picking images to be annotated and providing annotations for all tasks. In this paper, we show that it is more effective to select not only the images to be annotated but also a subset of tasks for which to provide annotations at each AL iteration. Furthermore, the annotations that are provided can be used to guess pseudo-labels for the tas…
University of California, Berkeley（加州大学伯克利分校）；Shanghai Jiao Tong University（上海交通大学）
Given the large-scale data and the high annotation cost, pretraining-finetuning becomes a popular paradigm in multiple computer vision tasks. Previous research has covered both the unsupervised pretraining and supervised finetuning in this paradigm, while little attention is paid to exploiting the annotation budget for finetuning. To fill in this gap, we formally define this new active finetuning task focusing on the selection of samples for annotation in the pretraining-finetuning paradigm. We propose a novel method called ActiveFT for active finetuning task to select a subset of data distrib…
We present a novel region based active learning method for semantic image segmentation, called MetaBox+. For acquisition, we train a meta regression model to estimate the segment-wise Intersection over Union (IoU) of each predicted segment of unlabeled images. This can be understood as an estimation of segment-wise prediction quality. Queried regions are supposed to minimize to competing targets, i.e., low predicted IoU values / segmentation quality and low estimated annotation costs. For estimating the latter we propose a simple but practical method for annotation cost estimation. We compare …
Indian Institute of Technology（印度理工学院）；The University of Tokyo（东京大学）
Active learning approaches in computer vision generally involve querying strong labels for data. However, previous works have shown that weak supervision can be effective in training models for vision tasks while greatly reducing annotation costs. Using this knowledge, we propose an adaptive supervision framework for active learning and demonstrate its effectiveness on the task of object detection. Instead of directly querying bounding box annotations (strong labels) for the most informative samples, we first query weak labels and optimize the model. Using a switching condition, the required s…
Huawei Inc.（华为公司）；Tsinghua University（清华大学）；Chinese Institute for Brain Research (CIBR)（中国脑科学研究所（北京脑科学与类脑研究中心））；IDG/McGovern Institute for Brain Research, Tsinghua University（清华大学IDG/麦戈文脑科学研究院）；BNRist（北京国家信息科学技术研究中心）
The requirement of expensive annotations is a major burden for training a well-performed instance segmentation model. In this paper, we present an economic active learning setting, named active pointly-supervised instance segmentation (APIS), which starts with box-level annotations and iteratively samples a point within the box and asks if it falls on the object. The key of APIS is to find the most desirable points to maximize the segmentation accuracy with limited annotation budgets. We formulate this setting and propose several uncertainty-based sampling strategies. The model developed with …
South China University of Technology（华南理工大学）；I2R, A*STAR（新加坡科技研究局资讯通信研究院）
Deep learning models are the state-of-the-art methods for semantic point cloud segmentation, the success of which relies on the availability of large-scale annotated datasets. However, it can be extremely time-consuming and prohibitively expensive to compile such datasets. In this work, we propose an active learning approach to maximize model performance given limited annotation budgets. We investigate the appropriate sample granularity for active selection under realistic annotation cost measurement (clicks), and demonstrate that super-point based selection allows for more efficient usage of …
Kyushu University（九州大学）；National Institute of Informatics（国立信息学研究所）；Kyoto Second Red Cross Hospital（京都第二红十字医院）
Automatic image-based disease severity estimation generally uses discrete (i.e., quantized) severity labels. Annotating discrete labels is often difficult due to the images with ambiguous severity. An easier alternative is to use relative annotation, which compares the severity level between image pairs. By using a learning-to-rank framework with relative annotation, we can train a neural network that estimates rank scores that are relative to severity levels. However, the relative annotation for all possible pairs is prohibitive, and therefore, appropriate sample pair selection is mandatory. …
Shanghai Jiao Tong University（上海交通大学）；University of California, Los Angeles（加州大学洛杉矶分校）；Chongqing University of Posts and Telecommunications（重庆邮电大学）
This paper presents a cost-sensitive active Question-Answering (QA) framework for learning a nine-layer And-Or graph (AOG) from web images. The AOG explicitly represents object categories, poses/viewpoints, parts, and detailed structures within the parts in a compositional hierarchy. The QA framework is designed to minimize an overall risk, which trades off the loss and query costs. The loss is defined for nodes in all layers of the AOG, including the generative loss (measuring the likelihood of the images) and the discriminative loss (measuring the fitness to human answers). The cost comprise…
IBM Research AI（IBM研究人工智能）；IBM Research | Africa（IBM研究|非洲）；CSAM at University of the Witwatersrand（威特沃特斯兰德大学CSAM）
An ML-based system for interactive labeling of image datasets is contributed in TensorBoard Projector to speed up image annotation performed by humans. The tool visualizes feature spaces and makes it directly editable by online integration of applied labels, and it is a system for verifying and managing machine learning data pertaining to labels. We propose realistic annotation emulation to evaluate the system design of interactive active learning, based on our improved semi-supervised extension of t-SNE dimensionality reduction. Our active learning tool can significantly increase labeling eff…
It is prohibitively expensive to annotate a large-scale video-based person re-identification (re-ID) dataset, which makes fully supervised methods inapplicable to real-world deployment. How to maximally reduce the annotation cost while retaining the re-ID performance becomes an interesting problem. In this paper, we address this problem by integrating an active learning scheme into a deep learning framework. Noticing that the truly matched tracklet-pairs, also denoted as true positives (TP), are the most informative samples for our re-ID model, we propose a sampling criterion to choose the mos…
State of the art methods for semantic image segmentation are trained in a supervised fashion using a large corpus of fully labeled training images. However, gathering such a corpus is expensive, due to human annotation effort, in contrast to gathering unlabeled data. We propose an active learning-based strategy, called CEREALS, in which a human only has to hand-label a few, automatically selected, regions within an unlabeled image corpus. This minimizes human annotation effort while maximizing the performance of a semantic image segmentation method. The automatic selection procedure is achieve…
University of Freiburg（弗赖堡大学）；German Aerospace Center (DLR)（德国航空航天中心（DLR））
Active learning is particularly of interest for semantic segmentation, where annotations are costly. Previous academic studies focused on datasets that are already very diverse and where the model is trained in a supervised manner with a large annotation budget. In contrast, data collected in many driving scenarios is highly redundant, and most medical applications are subject to very constrained annotation budgets. This work investigates the various types of existing active learning methods for semantic segmentation under diverse conditions across three dimensions - data distribution w.r.t. d…
Xi’an Jiaotong University（西安交通大学）；Pennsylvania State University（宾夕法尼亚州立大学）；the First Affiliated Hospital of Xi’an Jiaotong University（西安交通大学第一附属医院）
Digital pathology plays a crucial role in the development of artificial intelligence in the medical field. The digital pathology platform can make the pathological resources digital and networked, and realize the permanent storage of visual data and the synchronous browsing processing without the limitation of time and space. It has been widely used in various fields of pathology. However, there is still a lack of an open and universal digital pathology platform to assist doctors in the management and analysis of digital pathological sections, as well as the management and structured descripti…
Xidian University（西安电子科技大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
3D image segmentation is one of the most important and ubiquitous problems in medical image processing. It provides detailed quantitative analysis for accurate disease diagnosis, abnormal detection, and classification. Currently deep learning algorithms are widely used in medical image segmentation, most algorithms trained models with full annotated datasets. However, obtaining medical image datasets is very difficult and expensive, and full annotation of 3D medical image is a monotonous and time-consuming work. Partially labelling informative slices in 3D images will be a great relief of manu…
Martinos Center for Biomedical Imaging, Massachusetts General Hospital and Harvard Medical School（马蒂诺斯生物医学成像中心、麻省总医院、哈佛医学院）；Computer Science and Artificial Intelligence Laboratory (CSAIL), Massachusetts Institute of Technology（麻省理工学院计算机科学与人工智能实验室（CSAIL））；Wellcome / EPSRC Centre for Interventional and Surgical Sciences, University College London（伦敦大学学院惠康/EPSRC介入与外科科学中心）；Centre for Medical Image Computing, University College London（伦敦大学学院医学图像计算中心）；KU Leuven（鲁汶大学）；King’s College London（伦敦国王学院）
Video mosaicking requires the registration of overlapping frames located at distant timepoints in the sequence to ensure global consistency of the reconstructed scene. However, fully automated registration of such long-range pairs is (i) challenging when the registration of images itself is difficult; and (ii) computationally expensive for long sequences due to the large number of candidate pairs for registration. In this paper, we introduce an efficient framework for the active annotation of long-range pairwise correspondences in a sequence. Our framework suggests pairs of images that are sou…
Active Label Cleaning for Reliable Detection of Electron Dense Deposits in Transmission Electron Microscopy Images
School of Biomedical Engineering（生物医学工程学院）；Southern Medical University（南方医科大学）；College of Letters and Science（文理学院）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；School of Basic Medical Sciences（基础医学学院）；Department of Nephrology（肾内科）；Nanfang Hospital, Southern Medical University（南方医科大学南芳医院）
StepAL: Step-aware Active Learning for Cataract Surgical Videos
Johns Hopkins University（约翰霍普金斯大学）；Wilmer Eye Institute（威尔默眼科研究所）；Johns Hopkins University School of Medicine（约翰霍普金斯大学医学院）；Malone Center for Engineering in Healthcare（马洛中心医疗工程中心）
In this work, we introduce a novel framework that employs cluster annotation to boost active learning by reducing the number of human interactions required to train deep neural networks. Instead of annotating single samples individually, humans can also label clusters, producing a higher number of annotated samples with the cost of a small label error. Our experiments show that the proposed framework requires 82% and 87% less human interactions for CIFAR-10 and EuroSAT datasets respectively when compared with the fully-supervised training while maintaining similar performance on the test set.
Nanyang Technological University（南洋理工大学）；SenseTime Research（商汤科技研究院）；Beihang University（北京航空航天大学）；Shanghai AI Laboratory（上海人工智能实验室）
Large-scale datasets play a vital role in computer vision. But current datasets are annotated blindly without differentiation to samples, making the data collection inefficient and unscalable. The open question is how to build a mega-scale dataset actively. Although advanced active learning algorithms might be the answer, we experimentally found that they are lame in the realistic annotation scenario where out-of-distribution data is extensive. This work thus proposes a novel active learning framework for realistic dataset annotation. Equipped with this framework, we build a high-quality visio…
ActiveDC: Distribution Calibration for Active Finetuning
School of Software, Beihang University（北京航空航天大学软件学院）；Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）；State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（北京航空航天大学虚拟现实技术与系统国家重点实验室）
Arizona State University（亚利桑那州立大学）；Mayo Clinic（梅奥诊所）
The splendid success of convolutional neural networks (CNNs) in computer vision is largely attributable to the availability of massive annotated datasets, such as ImageNet and Places. However, in medical imaging, it is challenging to create such large annotated datasets, as annotating medical images is not only tedious, laborious, and time consuming, but it also demands costly, specialty-oriented skills, which are not easily accessible. To dramatically reduce annotation cost, this paper presents a novel method to naturally integrate active learning and transfer learning (fine-tuning) into a si…
We propose an Active Learning approach to training a segmentation classifier that exploits geometric priors to streamline the annotation process in 3D image volumes. To this end, we use these priors not only to select voxels most in need of annotation but to guarantee that they lie on 2D planar patch, which makes it much easier to annotate than if they were randomly distributed in the volume. A simplified version of this approach is effective in natural 2D images. We evaluated our approach on Electron Microscopy and Magnetic Resonance image volumes, as well as on natural images. Comparing our …
Dedan Kimathi University of Technology（德丹·基马蒂科技大学）
Digital data collected over the decades and data currently being produced with use of information technology is vastly the unlabeled data or data without description. The unlabeled data is relatively easy to acquire but expensive to label even with use of domain experts. Most of the recent works focus on use of active learning with uncertainty metrics measure to address this problem. Although most uncertainty selection strategies are very effective, they fail to take informativeness of the unlabeled instances into account and are prone to querying outliers. In order to address these challenges…
Acquiring the most representative examples via active learning (AL) can benefit many data-dependent computer vision tasks by minimizing efforts of image-level or pixel-wise annotations. In this paper, we propose a novel Collaborative Panoptic-Regional Active Learning framework (CPRAL) to address the semantic segmentation task. For a small batch of images initially sampled with pixel-wise annotations, we employ panoptic information to initially select unlabeled samples. Considering the class imbalance in the segmentation dataset, we import a Regional Gaussian Attention module (RGA) to achieve s…
Harvard University（哈佛大学）；University of Waterloo（滑铁卢大学）
Document images often have intricate layout structures, with numerous content regions (e.g. texts, figures, tables) densely arranged on each page. This makes the manual annotation of layout datasets expensive and inefficient. These characteristics also challenge existing active learning methods, as image-level scoring and selection suffer from the overexposure of common objects.Inspired by recent progresses in semi-supervised learning and self-training, we propose an Object-Level Active Learning framework for efficient document layout Annotation, OLALA. In this framework, only regions with the…
Embodied Active Learning under Limited Annotation and Navigation Budget for Object Detection
LAAS-CNRS, Universite de Toulouse, CNRS（法国国家科学研究中心图卢兹分部LAAS、图卢兹大学、法国国家科学研究中心）；Department of Electrical Engineering and Automation at Aalto University（阿尔托大学电气工程与自动化系）
Technical University of Munich（慕尼黑工业大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Recent advances in deep learning have resulted in great successes in various applications. Although semi-supervised or unsupervised learning methods have been widely investigated, the performance of deep neural networks highly depends on the annotated data. The problem is that the budget for annotation is usually limited due to the annotation time and expensive annotation cost in medical data. Active learning is one of the solutions to this problem where an active learner is designed to indicate which samples need to be annotated to effectively train a target model. In this paper, we propose a…
Indian Institute of Information Technology Allahabad（阿拉哈巴德印度信息技术学院）
In the clinical settings, during digital examination of histopathological slides, the pathologist annotate the slides by marking the rough boundary around the suspected tumour region. The marking or annotation is generally represented as a polygonal boundary that covers the extent of the tumour in the slide. These polygonal markings are difficult to imitate through CAD techniques since the tumour regions are heterogeneous and hence segmenting them would require exhaustive pixel wise ground truth annotation. Therefore, for CAD analysis, the ground truths are generally annotated by pathologist e…
