arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.09690cs.CLcs.LG
收藏

用途:提高语言模型的Few-Shot性能

Calibrate Before Use: Improving Few-Shot Performance of Language Models

UC Berkeley(加州大学伯克利分校);University of Maryland(马里兰大学);UC Irvine(加州大学欧文分校)

Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, Sameer Singh

摘要:

GPT-3在提供包含一些训练样本的自然语言提示时可以执行许多任务。我们证明了这种类型的少样本学习可能是不稳定的:提示格式的选择,训练样本,甚至训练样本的顺序都可能导致准确性从接近机会到接近最先进水平。我们证明了这种不稳定性源于语言模型对预测某些答案的偏见,例如,那些被放置在提示的末尾附近或者在预训练数据中是常见的。为了减轻这种情况,我们首先通过在给定训练提示和无内容测试输入(如“N/A”)时询问其预测来估计模型对每个答案的偏差。然后,我们拟合校准参数,使该输入的预测在答案中保持一致。在一组不同的任务中,这种上下文校准程序大大提高了GPT-3和GPT-2的平均准确度(高达30.0%的绝对值),并减少了提示的不同选择之间的差异。

英文摘要:

GPT-3 can perform numerous tasks when provided a natural language prompt that contains a few training examples. We show that this type of few-shot learning can be unstable: the choice of prompt format, training examples, and even the order of the training examples can cause accuracy to vary from near chance to near state-of-the-art. We demonstrate that this instability arises from the bias of language models towards predicting certain answers, e.g., those that are placed near the end of the prompt or are common in the pre-training data. To mitigate this, we first estimate the model's bias towards each answer by asking for its prediction when given the training prompt and a content-free test input such as "N/A". We then fit calibration parameters that cause the prediction for this input to be uniform across answers. On a diverse set of tasks, this contextual calibration procedure substantially improves GPT-3 and GPT-2's average accuracy (up to 30.0% absolute) and reduces variance across different choices of the prompt.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑