arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.14802cs.CL
收藏

无需标注测试数据的大语言模型能力估计

Estimating Large Language Model Capabilities without Labeled Test Data

University of Southern California(南加州大学)

Harvey Yiyun Fu, Qinyuan Ye, Albert Xu, Xiang Ren, Robin Jia

AI总结:

本文提出ICL准确率估计任务,利用LLM置信度分数训练元模型,在无需标注测试数据的情况下预测大语言模型上下文学习的准确率,并在多个模型和任务上验证其优于现有基线。

摘要:

大语言模型(LLMs)具有仅从少量示例中进行上下文学习(ICL)的出色能力,但ICL的成功率因任务不同而差异很大。因此,快速判断ICL是否适用于新任务非常重要,但在测试数据标注成本高昂的情况下——正是ICL最具吸引力的场景——直接评估ICL准确率可能代价很高。本文提出了ICL准确率估计任务,即仅给定新任务的未标注测试数据,预测LLM在该任务上进行上下文学习时的准确率。为实现ICL准确率估计,我们提出了一种方法,利用LLM置信度分数作为特征来训练元模型。我们在一个涵盖4个LLM和3个任务集合的新基准上,将我们的方法与多个强准确率估计基线进行了比较。元模型在12种设置中的8种上优于所有基线,并达到了与每个任务直接评估40个已标注测试示例相同的估计性能。同时,现有方法均无法在每种设置下提供准确可靠的ICL准确率估计,这凸显了需要更好的方法来衡量LLM预测的不确定性。

英文摘要:

Large Language Models (LLMs) have the impressive ability to perform in-context learning (ICL) from only a few examples, but the success of ICL varies widely from task to task. Thus, it is important to quickly determine whether ICL is applicable to a new task, but directly evaluating ICL accuracy can be expensive in situations where test data is expensive to annotate -- the exact situations where ICL is most appealing. In this paper, we propose the task of ICL accuracy estimation, in which we predict the accuracy of an LLM when doing in-context learning on a new task given only unlabeled test data for that task. To perform ICL accuracy estimation, we propose a method that trains a meta-model using LLM confidence scores as features. We compare our method to several strong accuracy estimation baselines on a new benchmark that covers 4 LLMs and 3 task collections. The meta-model improves over all baselines across 8 out of 12 settings and achieves the same estimation performance as directly evaluating on 40 collected labeled test examples per task. At the same time, no existing approach provides an accurate and reliable ICL accuracy estimation in every setting, highlighting the need for better ways to measure the uncertainty of LLM predictions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑