arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2112.08787cs.CL
收藏

AcTune:基于预训练语言模型的半监督主动学习的不确定性感知主动自我训练

AcTune: Uncertainty-aware Active Self-Training for Semi-Supervised Active Learning with Pretrained Language Models

Georgia Institute of Technology(佐治亚理工学院);University of Washington(华盛顿大学)

Yue Yu, Lingkai Kong, Jieyu Zhang, Rongzhi Zhang, Chao Zhang

摘要:

虽然预训练的语言模型(PLM)微调在许多NLP任务中取得了很好的性能,但微调阶段在标记数据中仍然要求很高。最近的工作已经采取主动微调,以提高PLM微调的标签效率,但他们都没有调查的潜力,未标记的数据。我们提出了{ \ours },一个新的框架,利用未标记的数据,以提高标签效率的主动PLM微调。AcTune基于不确定性在数据注释和模型自训练之间切换:它选择高不确定性的未标记样本进行主动注释,选择低不确定性的样本进行模型自训练。在这个框架下,我们设计了(1)一个区域感知的采样策略,减少冗余时,积极查询注释和(2)基于动量的存储体,动态聚合模型的伪标签,以抑制标签噪声在自我训练。在6个文本分类数据集上的实验表明,AcTune的性能优于最强的主动学习和自训练基线,并将PLM微调的标签效率平均提高了56.2%。我们的实现将在\url{https://github.com/yueyu1030/actune}上提供。

英文摘要:

While pre-trained language model (PLM) fine-tuning has achieved strong performance in many NLP tasks, the fine-tuning stage can be still demanding in labeled data. Recent works have resorted to active fine-tuning to improve the label efficiency of PLM fine-tuning, but none of them investigate the potential of unlabeled data. We propose {\ours}, a new framework that leverages unlabeled data to improve the label efficiency of active PLM fine-tuning. AcTune switches between data annotation and model self-training based on uncertainty: it selects high-uncertainty unlabeled samples for active annotation and low-uncertainty ones for model self-training. Under this framework, we design (1) a region-aware sampling strategy that reduces redundancy when actively querying for annotations and (2) a momentum-based memory bank that dynamically aggregates the model's pseudo labels to suppress label noise in self-training. Experiments on 6 text classification datasets show that AcTune outperforms the strongest active learning and self-training baselines and improves the label efficiency of PLM fine-tuning by 56.2\% on average. Our implementation will be available at \url{https://github.com/yueyu1030/actune}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑