arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.13274cs.CL
收藏

预算约束下的上下文学习:以词元分类为例

In-Context Learning on a Budget: A Case Study in Token Classification

School of Computer Science and Engineering, The Hebrew University of Jerusalem(耶路撒冷希伯来大学计算机科学与工程学院);School of Computing and Information Systems, University of Melbourne(墨尔本大学计算与信息系统学院);Microsoft Corporation(微软公司)

Uri Berger, Tal Baumel, Gabriel Stanovsky

AI总结:

本文研究在有限标注预算下的上下文学习,针对词元分类任务评估多种样本选择方法,发现各方法表现相近且小规模标注样本池即可达到与全量训练集相当的性能。

摘要:

少样本上下文学习(ICL)通常假设可以获取大型标注训练集。然而,在许多现实场景中,如领域自适应,只有有限的预算来标注少量样本,目标是最大化下游性能。我们研究了在预定义预算内选择样本进行标注的各种方法,重点关注词元分类任务,这些任务标注成本高昂且在ICL设置中研究较少。在各种任务、模型和数据集上,我们观察到没有方法显著优于其他方法,大多数方法产生相似的结果,包括随机样本选择进行标注。此外,我们证明相对较小的标注样本池可以达到与使用整个训练集相当的性能。我们希望未来的工作采用我们考虑标注预算的现实范式。

英文摘要:

Few shot in-context learning (ICL) typically assumes access to large annotated training sets. However, in many real world scenarios, such as domain adaptation, there is only a limited budget to annotate a small number of samples, with the goal of maximizing downstream performance. We study various methods for selecting samples to annotate within a predefined budget, focusing on token classification tasks, which are expensive to annotate and are relatively less studied in ICL setups. Across various tasks, models, and datasets, we observe that no method significantly outperforms the others, with most yielding similar results, including random sample selection for annotation. Moreover, we demonstrate that a relatively small annotated sample pool can achieve performance comparable to using the entire training set. We hope that future work adopts our realistic paradigm which takes annotation budget into account.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑