arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.00402cs.CLcs.AI
收藏

如果只需要检索,这真的算长上下文吗?迈向真正困难的长上下文自然语言处理

Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP

Bar-Ilan University(巴伊兰大学)

Omer Goldman, Alon Jacovi, Aviv Slobodkin, Aviya Maimon, Ido Dagan, Reut Tsarfaty

AI总结:

本文提出用扩散度和范围两个正交维度重新定义长上下文任务难度,指出真正困难且信息高度扩散的长上下文场景尚未被充分探索,并呼吁设计更具区分度的长上下文基准。

摘要:

语言模型能力的提升推动了其应用向更长上下文发展,使得长上下文评估与开发成为一个活跃的研究领域。然而,许多截然不同的用例被统一归入“长上下文”这一笼统术语之下,该术语仅由模型输入的总长度来定义,例如包括“大海捞针”(Needle-in-a-Haystack)任务、书籍摘要和信息聚合等。鉴于这些任务的难度各异,在这篇立场论文中我们认为,仅根据上下文长度将不同任务混为一谈是没有成效的。作为一个研究社区,我们需要一套更精确的词汇来理解是什么使得长上下文任务彼此相似或不同。我们提议基于那些使任务在更长上下文中变得更困难的性质,对长上下文进行分类解构。我们提出两个正交的难度轴:(I)扩散度(Diffusion):在上下文中找到必要信息有多困难?(II)范围(Scope):需要找到的必要信息有多少?我们综述了长上下文领域的文献,为这一分类法作为信息性描述符提供了论证,并据此对现有文献进行了定位。我们得出结论:最困难且最有趣的设置,即必要信息非常长且在输入中高度扩散的设置,目前被严重探索不足。通过使用描述性词汇并讨论长上下文中困难的相关性质,我们可以在此领域开展更具信息量的研究。我们呼吁精心设计具有明显长上下文的任务和基准,并充分考虑使其在性质上区别于较短上下文的特征。

英文摘要:

Improvements in language models' capabilities have pushed their applications towards longer contexts, making long-context evaluation and development an active research area. However, many disparate use-cases are grouped together under the umbrella term of "long-context", defined simply by the total length of the model's input, including - for example - Needle-in-a-Haystack tasks, book summarization, and information aggregation. Given their varied difficulty, in this position paper we argue that conflating different tasks by their context length is unproductive. As a community, we require a more precise vocabulary to understand what makes long-context tasks similar or different. We propose to unpack the taxonomy of long-context based on the properties that make them more difficult with longer contexts. We propose two orthogonal axes of difficulty: (I) Diffusion: How hard is it to find the necessary information in the context? (II) Scope: How much necessary information is there to find? We survey the literature on long-context, provide justification for this taxonomy as an informative descriptor, and situate the literature with respect to it. We conclude that the most difficult and interesting settings, whose necessary information is very long and highly diffused within the input, is severely under-explored. By using a descriptive vocabulary and discussing the relevant properties of difficulty in long-context, we can implement more informed research in this area. We call for a careful design of tasks and benchmarks with distinctly long context, taking into account the characteristics that make it qualitatively different from shorter context.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑