论模拟主动学习的局限性
On the Limitations of Simulating Active Learning
AI总结:
本文探讨了模拟主动学习(AL)的局限性,指出在已标注数据集上评估AL算法可能仅提供其真实有效性的下界,并分析了实验设置中的陷阱,为未来研究提供了指南。
摘要:
主动学习(AL)是一种人与模型在环的范式,通过迭代地选择信息丰富的未标注数据供人类标注,旨在改进随机采样。然而,实时使用人类标注执行AL实验是一个费力且昂贵的过程,因此对学术研究而言不切实际。解决这一障碍的简单方法是将已标注且公开可用的数据集视为未标注数据池,以此模拟AL。在这篇立场文章中,我们首先调查了近期文献,并强调了AL循环内所有不同步骤中的挑战。我们进一步揭示了实验设置中被忽视的陷阱,这些陷阱可能显著影响AL研究的质量。我们继续探讨模拟设置如何支配实证发现,认为它可能是“为什么主动学习算法有时无法超越随机采样?”这一常被提出的问题背后的答案之一。我们认为,在可用标注数据集上评估AL算法可能仅提供其在真实数据中有效性的下界。我们相信,共同塑造AL研究的最佳实践至关重要,特别是随着LLM工程技术的进步推动研究重心转向数据驱动方法(例如,数据效率、对齐、公平性)。有鉴于此,我们为未来工作制定了指南。我们的目标是引起社区对这些局限性的关注,以期找到解决它们的方法。
英文摘要:
Active learning (AL) is a human-and-model-in-the-loop paradigm that iteratively selects informative unlabeled data for human annotation, aiming to improve over random sampling. However, performing AL experiments with human annotations on-the-fly is a laborious and expensive process, thus unrealistic for academic research. An easy fix to this impediment is to simulate AL, by treating an already labeled and publicly available dataset as the pool of unlabeled data. In this position paper, we first survey recent literature and highlight the challenges across all different steps within the AL loop. We further unveil neglected caveats in the experimental setup that can significantly affect the quality of AL research. We continue with an exploration of how the simulation setting can govern empirical findings, arguing that it might be one of the answers behind the ever posed question ``why do active learning algorithms sometimes fail to outperform random sampling?''. We argue that evaluating AL algorithms on available labeled datasets might provide a lower bound as to their effectiveness in real data. We believe it is essential to collectively shape the best practices for AL research, particularly as engineering advancements in LLMs push the research focus towards data-driven approaches (e.g., data efficiency, alignment, fairness). In light of this, we have developed guidelines for future work. Our aim is to draw attention to these limitations within the community, in the hope of finding ways to address them.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。