arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2309.06131cs.IRcs.CL
收藏

为微调神经排序器标注数据?当前主动学习策略并不优于随机选择

Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random Selection

TU Wien(维也纳工业大学);University of Queensland(昆士兰大学);Cohere;Leiden University(莱顿大学)

Sophia Althammer, Guido Zuccon, Sebastian Hofstätter, Suzan Verberne, Allan Hanbury

AI总结:

本文研究了在有限数据和预算下微调PLM排序器,发现现有主动学习策略在有效性上不优于随机选择,且在固定标注成本下表现更差,无法识别低成本高效的最优数据子集。

摘要:

基于预训练语言模型(PLM)的搜索方法相较于统计模型和早期神经排序模型展现出巨大的有效性提升。然而,微调基于PLM的排序器需要大量标注训练数据。数据标注涉及大量人工工作,因此成本高昂,尤其是在特定领域任务中。本文在有限的训练数据和预算下研究基于PLM排序器的微调。我们研究了两种场景:从零开始微调排序器,以及从一个已在通用数据上微调过的排序器开始进行领域适应,并在目标数据集上继续微调。我们观察到,在不同的随机选择训练数据子集上进行微调时,有效性存在巨大差异。这表明,通过主动选择对排序器具有最积极影响的训练数据子集,有可能实现有效性提升。这样,就有可能以较低的标注预算微调出有效的PLM排序器。为了研究这一点,我们将现有的主动学习(AL)策略应用于微调PLM排序器的任务,并研究其有效性,同时考虑标注和计算成本。我们的广泛分析表明,在有效性方面,AL策略并未显著优于训练子集的随机选择。我们进一步发现,AL策略带来的收益是以更多评估(因此更高的标注成本)为代价的,并且在固定标注成本下比较有效性时,AL策略的表现不如随机选择。我们的结果强调,确实存在以低标注成本提供高有效性的“最优”训练数据子集,但应用于PLM排序器的当前主流AL策略无法识别它们。

英文摘要:

Search methods based on Pretrained Language Models (PLM) have demonstrated great effectiveness gains compared to statistical and early neural ranking models. However, fine-tuning PLM-based rankers requires a great amount of annotated training data. Annotating data involves a large manual effort and thus is expensive, especially in domain specific tasks. In this paper we investigate fine-tuning PLM-based rankers under limited training data and budget. We investigate two scenarios: fine-tuning a ranker from scratch, and domain adaptation starting with a ranker already fine-tuned on general data, and continuing fine-tuning on a target dataset. We observe a great variability in effectiveness when fine-tuning on different randomly selected subsets of training data. This suggests that it is possible to achieve effectiveness gains by actively selecting a subset of the training data that has the most positive effect on the rankers. This way, it would be possible to fine-tune effective PLM rankers at a reduced annotation budget. To investigate this, we adapt existing Active Learning (AL) strategies to the task of fine-tuning PLM rankers and investigate their effectiveness, also considering annotation and computational costs. Our extensive analysis shows that AL strategies do not significantly outperform random selection of training subsets in terms of effectiveness. We further find that gains provided by AL strategies come at the expense of more assessments (thus higher annotation costs) and AL strategies underperform random selection when comparing effectiveness given a fixed annotation cost. Our results highlight that ``optimal'' subsets of training data that provide high effectiveness at low annotation cost do exist, but current mainstream AL strategies applied to PLM rankers are not capable of identifying them.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑