arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.05136cs.CL
收藏

LV-Eval: 一个涵盖最高256K的5种长度层级的均衡长上下文基准

LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K

Tsinghua University(清华大学);Infinigence-AI;Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Shanghai Jiao Tong University(上海交通大学)

Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, Guohao Dai, Shengen Yan, Yu Wang

AI总结:

针对现有长上下文基准长度不足、存在知识泄漏和指标偏差的问题,本文提出含5种长度层级(最高256k)的双语长上下文基准LV-Eval,通过三项关键技术优化评估,在15个LLM上验证了其有效性与合理性。

摘要:

当前最先进的大语言模型(LLM)宣称已支持256k甚至更长的上下文长度。相比之下,主流基准的平均上下文长度仅为5k-21k,存在不足,且受潜在知识泄漏和指标不准确问题困扰,导致评估存在偏差。本文提出LV-Eval,这是一个具有挑战性的长上下文基准,包含16k、32k、64k、128k和256k共5种长度层级,最长可达256k词。LV-Eval包含单跳问答和多跳问答两大核心任务,由11个双语数据集组成。LV-Eval的设计融入了三项关键技术,分别是混淆事实插入、关键词与短语替换,以及基于关键词召回的指标设计。LV-Eval的优势包括:可跨不同上下文长度进行可控评估、包含带有混淆事实的高难度测试用例、缓解了知识泄漏问题、评估结果更客观。我们在LV-Eval上评估了15个大语言模型,并针对基准构建技术开展了消融研究。结果表明:(i) Moonshot-v1以及近期的大规模开源模型(如Qwen-2.5-72B和Llama-3.1-70B)在LV-Eval上表现最优,尤其是在64k以下的长度区间。(ii) 不同模型的得分变化趋势存在明显差异。例如GLM-4-9B-128k、Yi-6B-200k和Llama3-8B-1M的性能下降相对平缓,但它们的绝对性能未必高于上下文长度更短的大语言模型。(iii) 当存在混淆信息时,大语言模型的性能会显著下降,在“大海捞针”压力测试中尤为明显。(iv) 知识泄漏和指标不准确相关问题会给评估带来偏差,而LV-Eval缓解了这些问题。所有数据集和评估代码已发布于:https://github.com/infinigence/LVEval。

英文摘要:

State-of-the-art large language models (LLMs) are now claiming remarkable supported context lengths of 256k or even more. In contrast, the average context lengths of mainstream benchmarks are insufficient (5k-21k), and they suffer from potential knowledge leakage and inaccurate metrics, resulting in biased evaluation. This paper introduces LV-Eval, a challenging long-context benchmark with five length levels (16k, 32k, 64k, 128k, and 256k) reaching up to 256k words. LV-Eval features two main tasks, single-hop QA and multi-hop QA, comprising 11 bilingual datasets. The design of LV-Eval has incorporated three key techniques, namely confusing facts insertion, keyword and phrase replacement, and keyword-recall-based metric design. The advantages of LV-Eval include controllable evaluation across different context lengths, challenging test instances with confusing facts, mitigated knowledge leakage, and more objective evaluations. We evaluate 15 LLMs on LV-Eval and conduct ablation studies on the benchmarking techniques. The results reveal that: (i) Moonshot-v1 and recent large-scale open-source models, such as Qwen-2.5-72B and Llama-3.1-70B, achieve the highest performance on LV-Eval, particularly at lengths below 64k. (ii) Models exhibit distinct score trends. For example, GLM-4-9B-128k, Yi-6B-200k, and Llama3-8B-1M exhibit a relatively gentle degradation of performance, but their absolute performances may not necessarily be higher than those of LLMs with shorter context lengths. (iii) LLMs' performances can significantly degrade in the presence of confusing information, especially in the pressure test of "needle in a haystack". (iv) Issues related to knowledge leakage and inaccurate metrics introduce bias in evaluation, and these concerns are alleviated in LV-Eval. All datasets and evaluation codes are released at: https://github.com/infinigence/LVEval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑