实施主动学习的实际障碍
Practical Obstacles to Deploying Active Learning
摘要:
主动学习(AL)是一种广泛使用的训练策略,用于在固定注释预算的情况下最大化预测性能。在AL中,迭代地选择用于注释的训练样本,通常是当前模型最不确定的那些样本(通过某种度量)。希望是主动采样导致更好的性能比独立同分布(i.i.d.)随机抽样虽然AL在回顾性评价中显示出了希望,但这些研究往往忽视了其使用的实际障碍。在本文中,我们表明,虽然AL可以提供的好处时,使用特定的模型和特定的域,目前的方法的好处不可靠的模型和任务的推广。这是有问题的,因为在实践中,人们没有机会探索和比较替代AL策略。此外,AL将训练数据集与用于指导其获取的模型相结合。我们发现,随后用主动获取的数据集训练后继模型并不总是优于i.i.d.上的训练。采样数据我们的研究结果提出了一个问题,即AL固有的缺点是否值得它往往提供的适度和不一致的性能增益。
英文摘要:
Active learning (AL) is a widely-used training strategy for maximizing predictive performance subject to a fixed annotation budget. In AL one iteratively selects training examples for annotation, often those for which the current model is most uncertain (by some measure). The hope is that active sampling leads to better performance than would be achieved under independent and identically distributed (i.i.d.) random samples. While AL has shown promise in retrospective evaluations, these studies often ignore practical obstacles to its use. In this paper we show that while AL may provide benefits when used with specific models and for particular domains, the benefits of current approaches do not generalize reliably across models and tasks. This is problematic because in practice one does not have the opportunity to explore and compare alternative AL strategies. Moreover, AL couples the training dataset with the model used to guide its acquisition. We find that subsequently training a successor model with an actively-acquired dataset does not consistently outperform training on i.i.d. sampled data. Our findings raise the question of whether the downsides inherent to AL are worth the modest and inconsistent performance gains it tends to afford.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。