arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.14726cs.CLcs.AI
收藏

基于交叉熵差分的上下文示例选择方法

In-Context Demonstration Selection with Cross Entropy Difference

Microsoft Cognitive Service Research(微软认知服务研究院)

Dan Iter, Reid Pryzant, Ruochen Xu, Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu

AI总结:

本文提出交叉熵差分(CED)方法,通过参数高效微调小型模型计算测试示例与候选示例的交叉熵差分来排序和选择上下文示例,在8个基准上验证其能提升多种LLMs的文本生成性能。

摘要:

大型语言模型(LLMs)可以利用上下文示例(in-context demonstrations)来提升零样本任务的表现。然而,选择最优的上下文示例具有挑战性,因为模型性能会因所选示例的不同而产生很大差异。我们提出了一种用于选择上下文示例的交叉熵差分(cross-entropy difference, CED)方法。我们的方法基于以下观察:上下文示例的有效性与测试示例在基于该示例微调后的语言模型上的困惑度(perplexity)呈负相关。我们利用参数高效微调(parameter efficient finetuning)在训练数据上训练小型模型,用于计算测试示例与每个候选上下文示例之间的交叉熵差分。该度量指标被用于针对每个测试输入独立地对上下文示例进行排序和选择。我们在一个混合领域数据集上评估了我们的方法,该数据集结合了8个基准,代表4类文本生成任务,结果表明用于上下文示例选择的CED方法能够提升多种LLMs的性能。

英文摘要:

Large language models (LLMs) can use in-context demonstrations to improve performance on zero-shot tasks. However, selecting the best in-context examples is challenging because model performance can vary widely depending on the selected examples. We present a cross-entropy difference (CED) method for selecting in-context demonstrations. Our method is based on the observation that the effectiveness of in-context demonstrations negatively correlates with the perplexity of the test example by a language model that was finetuned on that demonstration. We utilize parameter efficient finetuning to train small models on training data that are used for computing the cross-entropy difference between a test example and every candidate in-context demonstration. This metric is used to rank and select in-context demonstrations independently for each test input. We evaluate our method on a mix-domain dataset that combines 8 benchmarks, representing 4 text generation tasks, showing that CED for in-context demonstration selection can improve performance for a variety of LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑