arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.02028cs.CLcs.AIcs.IRcs.LG
收藏

为什么情境学习有时会失败?在开放式与封闭式问题上评估情境学习

Why does in-context learning fail sometimes? Evaluating in-context learning on open and closed questions

University of Southern California(南加州大学);Information Sciences Institute(信息科学研究所)

Xiang Li, Haoran Tang, Siyu Chen, Ziwei Wang, Ryan Chen, Marcin Abram

AI总结:

本文构建高难度科学问题基准,评估任务新颖性、难度及上下文相关性对开放式和封闭式问题中情境学习表现的影响,发现更相关上下文未必更有益。

摘要:

我们将情境学习(in-context learning)的表现作为任务新颖性和难度的函数,针对开放式与封闭式问题进行衡量。为此,我们创建了一个由高难度科学问题组成的新型基准,每个问题都配有不同相关程度的上下文。我们反直觉地发现,与主题更契合的上下文并不总是比较不相关的上下文更有帮助。这种效应在开放式问题以及高难度或高新颖性问题上尤为明显。该结果揭示了大型语言模型在处理封闭式与开放式问题之间的根本差异,并表明有必要针对多种不同类型的问题,对情境学习进行更稳健的评估。它还提出了一个新问题:如何为大型语言模型最优地选择上下文,尤其是在检索增强生成(Retrieval Augmented Generation,RAG)系统中。我们的结果表明,这一问题的答案可能高度依赖应用,并可能取决于多种因素,包括问题格式、所感知到的问题难度,以及我们所寻求信息的新颖性或流行程度。

英文摘要:

We measure the performance of in-context learning as a function of task novelty and difficulty for open and closed questions. For that purpose, we created a novel benchmark consisting of hard scientific questions, each paired with a context of various relevancy. We show that counter-intuitively, a context that is more aligned with the topic does not always help more than a less relevant context. This effect is especially visible for open questions and questions of high difficulty or novelty. This result reveals a fundamental difference between the treatment of close-form and open-form questions by large-language models and shows a need for a more robust evaluation of in-context learning on the variety of different types of questions. It also poses a new question of how to optimally select a context for large language models, especially in the context of Retrieval Augmented Generation (RAG) systems. Our results suggest that the answer to this question can be highly application-dependent and might be contingent on factors including the format of the question, the perceived difficulty level of the questions, and the novelty or popularity of the information we seek.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑