$k$ NN推理:基于无校正最近邻推理的超上下文学习
$k$NN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inference
摘要:
上下文学习(ICL)是一种以情境演示为条件的快速完成目标任务的学习方法,已成为LLM的主要应用。在本文中,我们首先揭示了一个实际的困境,这种典型的使用,它不能扩大与训练数据由于上下文长度的限制。此外,现有的工作表明ICL也存在各种偏差,需要细致的校准处理。为了解决这两个挑战,我们提倡一个简单而有效的解决方案,$k$ NN预测,它首先查询LLM与训练数据的分布式表示,然后预测测试实例,通过简单地参考最近的邻居。我们进行了全面的实验来证明其双重优势:1)免校准:$k$ NN验证不会直接将LLM输出分布与特定于任务的标签空间对齐,而是利用这种分布来对齐测试和训练实例。在可比较的Few-Shot场景下,它显著优于最先进的基于校准的方法。2)超越语境:$k$神经网络训练可以进一步有效地扩展尽可能多的训练数据,不断带来实质性的改进。缩放趋势在10个数量级范围内保持,从2次到1024次,以及从0.8B到30B的不同LLM尺度。它成功地将数据扩展连接到模型扩展,并为LLM部署的无梯度范例带来了新的潜力。代码是公开的。
英文摘要:
In-Context Learning (ICL), which formulates target tasks as prompt completion conditioned on in-context demonstrations, has become the prevailing utilization of LLMs. In this paper, we first disclose an actual predicament for this typical usage that it can not scale up with training data due to context length restriction. Besides, existing works have shown that ICL also suffers from various biases and requires delicate calibration treatment. To address both challenges, we advocate a simple and effective solution, $k$NN Prompting, which first queries LLM with training data for distributed representations, then predicts test instances by simply referring to nearest neighbors. We conduct comprehensive experiments to demonstrate its two-fold superiority: 1) Calibration-Free: $k$NN Prompting does not directly align LLM output distribution with task-specific label space, instead leverages such distribution to align test and training instances. It significantly outperforms state-of-the-art calibration-based methods under comparable few-shot scenario. 2) Beyond-Context: $k$NN Prompting can further scale up effectively with as many training data as are available, continually bringing substantial improvements. The scaling trend holds across 10 orders of magnitude ranging from 2 shots to 1024 shots as well as different LLMs scales ranging from 0.8B to 30B. It successfully bridges data scaling into model scaling, and brings new potentials for the gradient-free paradigm of LLM deployment. Code is publicly available.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。