自适应上下文学习:面向上下文示例选择与排序的信息压缩视角
Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering
AI总结:
针对当前ICL随机采样上下文示例的问题,本文提出自适应ICL原则与先选后排通用框架,在8个NLP数据集上实现40%相对提升,展现出缩小ICL与微调差距的潜力。
摘要:
尽管上下文学习(in-context learning, ICL)的少样本表现令人惊艳,但目前仍普遍采用随机采样示例作为上下文的做法。本文提出了一种ICL新原则:自适应上下文学习。该自适应机制旨在帮助每个样本找到一个能推导出正确预测的上下文示例排列(即选择与排序),从而最大化性能。为验证自适应ICL的有效性,我们提出了一种通用的“先选后排”框架,并采用新的选择与排序算法对其进行实例化。在8个不同的NLP数据集上开展的广泛评估显示,我们的自适应ICL方法相比常规实践设置实现了40%的相对提升。进一步分析揭示了自适应ICL的巨大潜力:若采用更先进的算法,它或许能够缩小ICL与微调之间的差距。我们已公开代码以推动该领域的未来研究,代码地址:https://github.com/Shark-NLP/self-adaptive-ICL
英文摘要:
Despite the surprising few-shot performance of in-context learning (ICL), it is still a common practice to randomly sample examples to serve as context. This paper advocates a new principle for ICL: self-adaptive in-context learning. The self-adaption mechanism is introduced to help each sample find an in-context example permutation (i.e., selection and ordering) that can derive the correct prediction, thus maximizing performance. To validate the effectiveness of self-adaptive ICL, we propose a general select-then-rank framework and instantiate it with new selection and ranking algorithms. Upon extensive evaluation on eight different NLP datasets, our self-adaptive ICL method achieves a 40% relative improvement over the common practice setting. Further analysis reveals the enormous potential of self-adaptive ICL that it might be able to close the gap between ICL and finetuning given more advanced algorithms. Our code is released to facilitate future research in this area: https://github.com/Shark-NLP/self-adaptive-ICL
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。