Similarity Search for Efficient Active Learning and Search of Rare Concepts
Stanford University（斯坦福大学）；Facebook AI（脸书人工智能）；University of Wisconsin（威斯康星大学）；Facebook AI Research（脸书人工智能研究院）
Cody Coleman, Edward Chou, Julian Katz-Samuels, Sean Culatana, Peter Bailis, Alexander C. Berg, Robert Nowak, Roshan Sumbaly, Matei Zaharia, I. Zeki Yalniz
Many active learning and search approaches are intractable for large-scale industrial settings with billions of unlabeled examples. Existing approaches search globally for the optimal examples to label, scaling linearly or even quadratically with the unlabeled data. In this paper, we improve the computational efficiency of active learning and search methods by restricting the candidate pool for labeling to the nearest neighbors of the currently labeled set instead of scanning over all of the unlabeled data. We evaluate several selection strategies in this setting on three large-scale computer vision datasets: ImageNet, OpenImages, and a de-identified and aggregated dataset of 10 billion images provided by a large internet company. Our approach achieved similar mean average precision and recall as the traditional global approach while reducing the computational cost of selection by up to three orders of magnitude, thus enabling web-scale active learning.
RareSpot+: A Benchmark, Model, and Active Learning Framework for Small and Rare Wildlife in Aerial Imagery
University of California, Santa Barbara（加州大学圣芭芭拉分校）；Smithsonian National Zoo and Conservation Biology Institute（史密森尼国家动物园与保护生物学研究所）；Stanford University（斯坦福大学）；The Doris Duke Foundation（多里斯·杜克基金会）
ActiveSSF: An Active-Learning-Guided Self-Supervised Framework for Long-Tailed Megakaryocyte Classification
School of Software Engineering, Xinjiang University（新疆大学软件工程学院）；Department of Hematology, Xiangya Hospital, Central South University（中南大学湘雅医院血液科）；National Clinical Research Center for Geriatric Diseases, Xiangya Hospital（中南大学湘雅医院老年病临床医学研究中心）；Hunan Hematology Oncology Clinical Medical Research Center（湖南省血液病肿瘤临床医学研究中心）；School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件工程学院）
INRIA Lille（里尔法国国家信息与自动化研究所）；Sapienza University of Rome（罗马大学）；Google（谷歌）
We investigate active learning by pairwise similarity over the leaves of trees originating from hierarchical clustering procedures. In the realizable setting, we provide a full characterization of the number of queries needed to achieve perfect reconstruction of the tree cut. In the non-realizable setting, we rely on known important-sampling procedures to obtain regret and query complexity bounds. Our algorithms come with theoretical guarantees on the statistical error and, more importantly, lend themselves to linear-time implementations in the relevant parameters of the problem. We discuss su…
Berkeley Center for Cosmological Physics（伯克利宇宙物理中心）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）；University of California, Berkeley（加州大学伯克利分校）
We present the use of self-supervised learning to explore and exploit large unlabeled datasets. Focusing on 42 million galaxy images from the latest data release of the Dark Energy Spectroscopic Instrument (DESI) Legacy Imaging Surveys, we first train a self-supervised model to distill low-dimensional representations that are robust to symmetries, uncertainties, and noise in each image. We then use the representations to construct and publicly release an interactive semantic similarity search tool. We demonstrate how our tool can be used to rapidly discover rare objects given only a single exa…
QC-PHAST Search: Classical--Quantum Query Benchmarks for Finite-Pool Rare-Regime Discovery
Department of Computer Science（计算机科学系）；College of Natural Sciences（自然科学学院）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Oden Institute for Computational Engineering and Sciences（计算工程与科学奥登研究所）
Active learning is commonly used to train label-efficient models by adaptively selecting the most informative queries. However, most active learning strategies are designed to either learn a representation of the data (e.g., embedding or metric learning) or perform well on a task (e.g., classification) on the data. However, many machine learning tasks involve a combination of both representation learning and a task-specific goal. Motivated by this, we propose a novel unified query framework that can be applied to any problem in which a key component is learning a representation of the data tha…
University Of Technology of Compiegne（贡比涅技术大学）；Chalmers University of Technology（查尔姆斯理工大学）；University of Gothenburg（哥德堡大学）；Zenseact
We describe an iterative active-learning algorithm to recognise rare traffic signs. A standard ResNet is trained on a training set containing only a single sample of the rare class. We demonstrate that by sorting the samples of a large, unlabeled set by the estimated probability of belonging to the rare class, we can efficiently identify samples from the rare class. This works despite the fact that this estimated probability is usually quite low. A reliable active-learning loop is obtained by labeling these candidate samples, including them in the training set, and iterating the procedure. Fur…
HPOQuest: A Rare-Disease Diagnostic Agent Using Active Phenotype Acquisition
Technical University of Munich（慕尼黑工业大学）；Helmholtz Center Munich（亥姆霍兹慕尼黑中心）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Munich Data Science Institute (MDSI)（慕尼黑数据科学研究所）；Aalto University（阿尔托大学）
School of Electrical and Computer Engineering（电气与计算机工程学院）；School of Interactive Computing（交互计算学院）；Georgia Institute of Technology（佐治亚理工学院）
Many machine learning tasks such as clustering, classification, and dataset search benefit from embedding data points in a space where distances reflect notions of relative similarity as perceived by humans. A common way to construct such an embedding is to request triplet similarity queries to an oracle, comparing two objects with respect to a reference. This work generalizes triplet queries to tuple queries of arbitrary size that ask an oracle to rank multiple objects against a reference, and introduces an efficient and robust adaptive selection method called InfoTuple that uses a novel appr…
Stanford University（斯坦福大学）；Carnegie Mellon University（卡内基梅隆大学）；Google（谷歌）；Argo AI
For machine learning models trained with limited labeled training data, validation stands to become the main bottleneck to reducing overall annotation costs. We propose a statistical validation algorithm that accurately estimates the F-score of binary classifiers for rare categories, where finding relevant examples to evaluate on is particularly challenging. Our key insight is that simultaneous calibration and importance sampling enables accurate estimates even in the low-sample regime (< 300 samples). Critically, we also derive an accurate single-trial estimator of the variance of our method …
Idaho National Laboratory（爱达荷国家实验室）；Johns Hopkins University（约翰斯·霍普金斯大学）
While multifidelity modeling provides a cost-effective way to conduct uncertainty quantification with computationally expensive models, much greater efficiency can be achieved by adaptively deciding the number of required high-fidelity (HF) simulations, depending on the type and complexity of the problem and the desired accuracy in the results. We propose a framework for active learning with multifidelity modeling emphasizing the efficient estimation of rare events. Our framework works by fusing a low-fidelity (LF) prediction with an HF-inferred correction, filtering the corrected LF predictio…
Active learning has proven to be useful for minimizing labeling costs by selecting the most informative samples. However, existing active learning methods do not work well in realistic scenarios such as imbalance or rare classes, out-of-distribution data in the unlabeled set, and redundancy. In this work, we propose SIMILAR (Submodular Information Measures based actIve LeARning), a unified active learning framework using recently proposed submodular information measures (SIM) as acquisition functions. We argue that SIMILAR not only works in standard active learning, but also easily extends to …
We present an active learning strategy for training parametric models of distance metrics, given triplet-based similarity assessments: object $x_i$ is more similar to object $x_j$ than to $x_k$. In contrast to prior work on class-based learning, where the fundamental goal is classification and any implicit or explicit metric is binary, we focus on {\em perceptual} metrics that express the {\em degree} of (dis)similarity between objects. We find that standard active learning approaches degrade when annotations are requested for {\em batches} of triplets at a time: our studies suggest that corre…
Johns Hopkins University（约翰斯·霍普金斯大学）；Idaho National Laboratory（爱达荷国家实验室）
Estimating the probability of failure for complex real-world systems using high-fidelity computational models is often prohibitively expensive, especially when the probability is small. Exploiting low-fidelity models can make this process more feasible, but merging information from multiple low-fidelity and high-fidelity models poses several challenges. This paper presents a robust multi-fidelity surrogate modeling strategy in which the multi-fidelity surrogate is assembled using an active learning strategy using an on-the-fly model adequacy assessment set within a subset simulation framework …
Yandex；Lomonosov Moscow State University（罗蒙诺索夫莫斯科国立大学）；National Research University Higher School of Economics（高等经济研究大学）
Similarity graphs are an active research direction for the nearest neighbor search (NNS) problem. New algorithms for similarity graph construction are continuously being proposed and analyzed by both theoreticians and practitioners. However, existing construction algorithms are mostly based on heuristics and do not explicitly maximize the target performance measure, i.e., search recall. Therefore, at the moment it is not clear whether the performance of similarity graphs has plateaued or more effective graphs can be constructed with more theoretically grounded methods. In this paper, we introd…
University of Southern California（南加州大学）；Télécom ParisTech（巴黎高等电信学院）；CNRS（法国国家科学研究中心）
A good measure of similarity between data points is crucial to many tasks in machine learning. Similarity and metric learning methods learn such measures automatically from data, but they do not scale well respect to the dimensionality of the data. In this paper, we propose a method that can learn efficiently similarity measure from high-dimensional sparse data. The core idea is to parameterize the similarity measure as a convex combination of rank-one matrices with specific sparsity structures. The parameters are then optimized with an approximate Frank-Wolfe procedure to maximally satisfy re…
Active Learning of Ordinal Embeddings: A User Study on Football Data
Friedrich-Alexander-Universität Erlangen-Nürnberg（弗里德里希-亚历山大-埃尔兰根-纽伦堡大学）；Fraunhofer Institute for Integrated Circuits（弗劳恩霍夫集成电路研究所）；Georgia Institute of Technology（佐治亚理工学院）
We consider the problem of wisely using a limited budget to label a small subset of a large unlabeled dataset. We are motivated by the NLP problem of word sense disambiguation. For any word, we have a set of candidate labels from a knowledge base, but the label set is not necessarily representative of what occurs in the data: there may exist labels in the knowledge base that very rarely occur in the corpus because the sense is rare in modern English; and conversely there may exist true labels that do not exist in our knowledge base. Our aim is to obtain a classifier that performs as well as po…
AnomalyMatch: Discovering Rare Objects of Interest with Semi-supervised and Active Learning
European Space Agency (ESA)（欧洲航天局）；European Space Astronomy Centre (ESAC)（欧洲空间天文学中心）；Astronomisches Rechen-Institut（天文计算研究所）；Zentrum für Astronomie der Universität Heidelberg（海德堡大学天文研究中心）；Kapteyn Astronomical Institute（Kapteyn天文研究所）；University of Groningen（格罗宁根大学）
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Loyola Marymount University（洛约拉马利蒙特大学）；University of Minnesota（明尼苏达大学）；Harvey Mudd College（哈维穆德学院）
We present a novel method for classification of Synthetic Aperture Radar (SAR) data by combining ideas from graph-based learning and neural network methods within an active learning framework. Graph-based methods in machine learning are based on a similarity graph constructed from the data. When the data consists of raw images composed of scenes, extraneous information can make the classification task more difficult. In recent years, neural network methods have been shown to provide a promising framework for extracting patterns from SAR images. These methods, however, require ample training da…
Visual-Interactive Similarity Search for Complex Objects by Example of Soccer Player Analysis
Technische Universität Darmstadt（达姆施塔特工业大学）；St. Pölten University of Applied Sciences（圣帕尔滕应用科学大学）；Fraunhofer Institute for Computer Graphics Research（弗劳恩霍夫计算机图形学研究所）
Washington University in St. Louis（圣路易斯华盛顿大学）；University of Toronto（多伦多大学）
Active search is a learning paradigm where we seek to identify as many members of a rare, valuable class as possible given a labeling budget. Previous work on active search has assumed access to a faithful (and expensive) oracle reporting experimental results. However, some settings offer access to cheaper surrogates such as computational simulation that may aid in the search. We propose a model of multifidelity active search, as well as a novel, computationally efficient policy for this setting that is motivated by state-of-the-art classical policies. Our policy is nonmyopic and budget aware,…
