arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.08559cs.CLcs.AI
收藏

大语言模型不是优秀的少样本文本信息抽取器,却是难样本的优秀重排器!

Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!

S-Lab, Nanyang Technological University(南洋理工大学S-Lab);Singapore Management University(新加坡管理大学)

Yubo Ma, Yixin Cao, YongChing Hong, Aixin Sun

AI总结:

该研究通过实验证明大语言模型少样本信息抽取性能弱于微调小模型,进而提出自适应过滤-重排范式,让小模型过滤、大模型重排难样本,以可接受成本提升抽取性能。

摘要:

大语言模型(LLMs)已在各类任务中取得显著进展。然而,LLMs在信息抽取(IE)任务中是否具备竞争力的少样本求解能力,仍是一个悬而未决的问题。本研究旨在为该问题提供全面的解答。通过在4类IE任务的9个数据集上开展大量实验,我们证明:在大多数设置下,当前先进的LLMs与经过微调的小语言模型(SLMs)相比,始终表现出性能更差、延迟更高、预算需求更大的特点。因此我们得出结论:总体而言LLMs并非有效的少样本信息抽取器。尽管如此,我们也说明,借助合适的提示策略,LLMs能够有效补充SLMs的能力,处理SLMs难以应对的困难样本。此外,我们提出一种自适应的“过滤-重排”范式,以结合LLMs与SLMs的优势。在该范式中,SLMs充当过滤器,LLMs充当重排器。通过提示LLMs对SLMs识别出的一小部分困难样本进行重排,我们的初步系统在各类IE任务上持续取得了可观的性能提升(平均F1值提升2.4%),且时间与成本投入处于可接受范围内。

英文摘要:

Large Language Models (LLMs) have made remarkable strides in various tasks. Whether LLMs are competitive few-shot solvers for information extraction (IE) tasks, however, remains an open problem. In this work, we aim to provide a thorough answer to this question. Through extensive experiments on nine datasets across four IE tasks, we demonstrate that current advanced LLMs consistently exhibit inferior performance, higher latency, and increased budget requirements compared to fine-tuned SLMs under most settings. Therefore, we conclude that LLMs are not effective few-shot information extractors in general. Nonetheless, we illustrate that with appropriate prompting strategies, LLMs can effectively complement SLMs and tackle challenging samples that SLMs struggle with. And moreover, we propose an adaptive filter-then-rerank paradigm to combine the strengths of LLMs and SLMs. In this paradigm, SLMs serve as filters and LLMs serve as rerankers. By prompting LLMs to rerank a small portion of difficult samples identified by SLMs, our preliminary system consistently achieves promising improvements (2.4% F1-gain on average) on various IE tasks, with an acceptable time and cost investment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑