让大语言模型物有所值:银行领域资源受限的文本分类
Making LLMs Worth Every Penny: Resource-Limited Text Classification in Banking
AI总结:
本研究在Banking77数据集上系统评估少样本与大语言模型方法的性能-成本权衡,提出基于RAG的低成本查询和GPT-4数据增强方法,并提供专家精选子集与错误分析以推动该领域研究。
摘要:
自然语言处理中的标准全量数据分类器需要数千个标注样本,这在数据有限的领域中是不切实际的。少样本方法提供了一种替代方案,利用对比学习技术,每类仅需20个样本即可有效工作。同样,像GPT-4这样的大语言模型(LLMs)每类仅需1-5个样本就能有效执行。然而,这些方法的性能-成本权衡仍未得到充分探索,这对预算受限的组织来说是一个关键问题。我们的工作通过研究上述方法在Banking77金融意图检测数据集上的表现来填补这一空白,包括在全面的少样本场景集合中评估OpenAI、Cohere和Anthropic的尖端大语言模型。我们通过两种额外的方法完善了这一图景:首先,一种基于检索增强生成(RAG)的具有成本效益的大语言模型查询方法,与经典少样本方法相比,能够将运营成本降低数倍;其次,一种使用GPT-4的数据增强方法,能够在数据受限场景中提升性能。最后,为了激发未来研究,我们提供了由人类专家精选的Banking77子集,以及广泛的错误分析。
英文摘要:
Standard Full-Data classifiers in NLP demand thousands of labeled examples, which is impractical in data-limited domains. Few-shot methods offer an alternative, utilizing contrastive learning techniques that can be effective with as little as 20 examples per class. Similarly, Large Language Models (LLMs) like GPT-4 can perform effectively with just 1-5 examples per class. However, the performance-cost trade-offs of these methods remain underexplored, a critical concern for budget-limited organizations. Our work addresses this gap by studying the aforementioned approaches over the Banking77 financial intent detection dataset, including the evaluation of cutting-edge LLMs by OpenAI, Cohere, and Anthropic in a comprehensive set of few-shot scenarios. We complete the picture with two additional methods: first, a cost-effective querying method for LLMs based on retrieval-augmented generation (RAG), able to reduce operational costs multiple times compared to classic few-shot approaches, and second, a data augmentation method using GPT-4, able to improve performance in data-limited scenarios. Finally, to inspire future research, we provide a human expert's curated subset of Banking77, along with extensive error analysis.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。