arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.07367cs.CL
收藏

基于大语言模型的自然语言处理主动学习

Active Learning for NLP with Large Language Models

Wayne State University(韦恩州立大学)

Xuesong Wang

AI总结:

本文研究用 GPT-3.5 和 GPT-4 替代人工标注,提出基于一致性的混合标注策略,在三个文本分类数据集上验证其能以更低成本达到与人工标注相近或更好的主动学习效果。

摘要:

训练样本的人工标注成本高昂、费时费力,有时甚至颇具挑战,对于自然语言处理(NLP)任务尤其如此。为了降低标注成本并提高样本效率,可以使用主动学习(Active Learning, AL)技术,以尽可能少的标注样本达到合理或相近的结果。为了进一步降低成本,并且随着大语言模型(Large Language Models, LLMs)的显著进步,LLMs 可以成为标注样本的良好候选者。本研究考察了使用 LLMs(GPT-3.5 和 GPT-4)在 3 个不同数据集上标注样本的准确性与成本。我们提出了一种基于一致性的策略,用于选择可能被错误标注的样本,从而在主动学习设置中由人工对这些样本进行标注,我们称之为混合标注策略。随后,我们在两种不同设置下测试了主动学习的性能:(1)仅使用人工标注;(2)使用所提出的混合标注策略。我们在 3 个文本分类数据集(即 AG's News、TREC-6 和 Rotten Tomatoes)上报告了主动学习模型在 3 种主动学习查询策略下的准确率。在 AG's News 和 Rotten Tomatoes 上,使用混合标注策略训练的模型取得了与使用人工标注训练的模型相似或更好的结果。该方法揭示了 LLMs 作为标注者在主动学习环境中在准确性和成本效率方面的巨大潜力。

英文摘要:

Human annotation of training samples is expensive, laborious, and sometimes challenging, especially for Natural Language Processing (NLP) tasks. To reduce the labeling cost and enhance the sample efficiency, Active Learning (AL) technique can be used to label as few samples as possible to reach a reasonable or similar results. To reduce even more costs and with the significant advances of Large Language Models (LLMs), LLMs can be a good candidate to annotate samples. This work investigates the accuracy and cost of using LLMs (GPT-3.5 and GPT-4) to label samples on 3 different datasets. A consistency-based strategy is proposed to select samples that are potentially incorrectly labeled so that human annotations can be used for those samples in AL settings, and we call it mixed annotation strategy. Then we test performance of AL under two different settings: (1) using human annotations only; (2) using the proposed mixed annotation strategy. The accuracy of AL models under 3 AL query strategies are reported on 3 text classification datasets, i.e., AG's News, TREC-6, and Rotten Tomatoes. On AG's News and Rotten Tomatoes, the models trained with the mixed annotation strategy achieves similar or better results compared to that with human annotations. The method reveals great potentials of LLMs as annotators in terms of accuracy and cost efficiency in active learning settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑