arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2101.08133cs.CL
收藏

使用深度预训练模型和贝叶斯不确定性估计进行序列标记的主动学习

Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates

Skolkovo Institute of Science and Technology(斯科尔科沃科技学院);HSE University(高等经济大学);Mobile TeleSystems (MTS)(移动电信系统公司);Federal Research Center “Computer Science and Control” of Russian Academy of Sciences(俄罗斯科学院计算机科学与控制联邦研究中心)

Artem Shelmanov, Dmitri Puzyrev, Lyubov Kupriyanova, Denis Belyakov, Daniil Larionov, Nikita Khromov, Olga Kozlova, Ekaterina Artemova, Dmitry V. Dylov, Alexander Panchenko

摘要:

注释用于文本序列标记的训练数据通常非常耗时。结合主动学习的自然语言处理的迁移学习的最新进展为显着减少必要的注释预算提供了可能性。我们是第一个彻底调查这个强大的组合序列标记任务。我们对主动学习框架中深度预训练模型的各种贝叶斯不确定性估计方法和蒙特卡洛退出选项进行了广泛的实证研究,并找到了不同类型模型的最佳组合。此外,我们还证明了在主动学习过程中获取实例,可以用蒸馏版本代替全尺寸的Transformer,这会产生更好的计算性能,并减少在实践中应用深度主动学习的障碍。

英文摘要:

Annotating training data for sequence tagging of texts is usually very time-consuming. Recent advances in transfer learning for natural language processing in conjunction with active learning open the possibility to significantly reduce the necessary annotation budget. We are the first to thoroughly investigate this powerful combination for the sequence tagging task. We conduct an extensive empirical study of various Bayesian uncertainty estimation methods and Monte Carlo dropout options for deep pre-trained models in the active learning framework and find the best combinations for different types of models. Besides, we also demonstrate that to acquire instances during active learning, a full-size Transformer can be substituted with a distilled version, which yields better computational performance and reduces obstacles for applying deep active learning in practice.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑