arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.15324cs.CVcs.AIcs.LG
收藏

深度主动学习在图像分类任务中的有效性评估

Effective Evaluation of Deep Active Learning on Image Classification Tasks

The University of Texas at Dallas(德克萨斯大学达拉斯分校);Indian Institute of Technology, Bombay(印度理工学院孟买分校);AIFY Innovation Labs(AIFY创新实验室)

Nathan Beck, Durga Sivasubramanian, Apurva Dani, Ganesh Ramakrishnan, Rishabh Iyer

摘要:

为了使深度学习更加标签有效,越来越多的论文一直在研究深度模型的主动学习(AL)。然而,在普遍的实验环境中存在一些问题,主要是由于缺乏统一的实施和基准。当前文献中的问题包括有时对不同AL算法性能的矛盾观察,无意中排除重要的泛化方法,如数据增强和SGD优化,缺乏对AL标记效率等评估方面的研究,以及AL优于随机采样(RS)的情况很少或没有明确。在这项工作中,我们提出了一个统一的重新实现的国家的最先进的AL算法的背景下,通过我们新的开源AL工具包DISTIL的图像分类,我们仔细研究这些问题的有效评估方面。在积极的一面,我们表明,AL技术是$2\times$到$4\times$更标签效率相比,RS与使用数据增强。令人惊讶的是,当包括数据增强时,使用BADGE(一种最先进的方法)与简单的不确定性采样相比不再有一致的增益。然后,我们仔细分析了现有的方法如何执行不同数量的冗余和每个类的例子。最后,我们提供了一些见解,供AL从业者在未来的工作中考虑,如AL批量大小的影响,初始化的影响,在每一轮重新训练模型的重要性,以及其他见解。

英文摘要:

With the goal of making deep learning more label-efficient, a growing number of papers have been studying active learning (AL) for deep models. However, there are a number of issues in the prevalent experimental settings, mainly stemming from a lack of unified implementation and benchmarking. Issues in the current literature include sometimes contradictory observations on the performance of different AL algorithms, unintended exclusion of important generalization approaches such as data augmentation and SGD for optimization, a lack of study of evaluation facets like the labeling efficiency of AL, and little or no clarity on the scenarios in which AL outperforms random sampling (RS). In this work, we present a unified re-implementation of state-of-the-art AL algorithms in the context of image classification via our new open-source AL toolkit DISTIL, and we carefully study these issues as facets of effective evaluation. On the positive side, we show that AL techniques are $2\times$ to $4\times$ more label-efficient compared to RS with the use of data augmentation. Surprisingly, when data augmentation is included, there is no longer a consistent gain in using BADGE, a state-of-the-art approach, over simple uncertainty sampling. We then do a careful analysis of how existing approaches perform with varying amounts of redundancy and number of examples per class. Finally, we provide several insights for AL practitioners to consider in future work, such as the effect of the AL batch size, the effect of initialization, the importance of retraining the model at every round, and other insights.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑