检索还是整体理解？Dolce：区分我们的长上下文评估任务
Retrieval Or Holistic Understanding? Dolce: Differentiate Our Long Context Evaluation Tasks
AI总结：
本文提出 Dolce 框架，通过短上下文采样与混合模型估计复杂度 λ 和冗余度 k，自动区分长上下文任务中的检索与整体理解类别并量化难度。
摘要：
我们认为，长上下文理解中存在两种主要且不同的能力：检索和整体理解。如果不了解任务所侧重的类别，就不可能理解并进一步提升大语言模型（LLM）的长上下文能力。我们的目标是从一系列基准测试中自动识别以检索为核心和以整体理解为核心的问题，并定量衡量每种侧重内部的难度。在本文中，我们提出了 Dolce 框架，该框架用 λ（复杂度）和 k（冗余度）对每个问题进行参数化，并将其归入五个预定义侧重类别之一。我们提议从完整上下文中采样短上下文，并估计 LLM 使用所采样片段解决问题的概率。为了找到每个问题的 λ 和 k，我们进一步提出了一个混合模型，由非参数背景噪声分量和参数化/非参数化混合 oracle 分量组成；在该模型中，我们针对正确或错误（COW）情形以及评分中的部分得分（PIG）情形，推导了由 λ 和 k 参数化的概率函数。在44个现有长上下文评估任务中，我们提出的方法能够识别出0%至67%的问题以检索为核心，以及0%至90%的问题以整体理解为核心。
英文摘要：
We argue that there are two major distinct capabilities in long context understanding: retrieval and holistic understanding. Understanding and further improving LLMs' long context capabilities would not be possible without knowing the tasks' focus categories. We aim to automatically identify retrieval focused and holistic understanding focused problems from suites of benchmarks and quantitatively measure the difficulty within each focus. In this paper, we present the Dolce framework, which parameterizes each problem by $λ$ (complexity) and $k$ (redundancy) and assigns to one of five predefined focus categories. We propose to sample short contexts from the full context and estimate the probability an LLM solves the problem using the sampled spans. To find the $λ$ and $k$ for each problem, we further propose a mixture model of a non-parametric background noise component and a parametric/non-parametric hybrid oracle component, where we derive the probability functions parameterized by $λ$ and $k$ for both the correct-or-wrong (COW) scenario and the partial-point-in-grading (PIG) scenario. Our proposed methods can identify 0% to 67% of the problems are retrieval focused and 0% to 90% of the problems are holistic understanding focused across 44 existing long context evaluation tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
