arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.17588cs.CL
收藏

LongIns:一项面向LLM的具有挑战性的长上下文指令测试

LongIns: A Challenging Long-context Instruction-based Exam for LLMs

M-A-P;University of Waterloo(滑铁卢大学)

Shawn Gavin, Tuney Zheng, Jiaheng Liu, Quehry Que, Noah Wang, Jian Yang, Chenchen Zhang, Wenhao Huang, Ge Zhang

AI总结:

提出LongIns基准,通过全局/局部指令与单/多任务设置评估LLMs长上下文能力,发现GPT-4在16k窗口表现不佳且现有模型在短窗口下多跳推理仍需改进。

摘要:

大型语言模型(LLMs)的长上下文能力近年来一直是热门话题。为了评估LLMs在不同场景下的性能,出现了各种评估基准。然而,由于这些基准大多侧重于识别关键信息来回答问题,这主要需要LLMs的检索能力,因此它们只能部分代表LLMs从大量信息中进行推理的性能。同时,尽管LLMs经常声称具有32k、128k、200k甚至更长的上下文窗口,但这些基准未能揭示这些LLMs实际支持的长度。为了解决这些问题,我们提出了LongIns基准数据集,这是一项面向LLMs的具有挑战性的长上下文指令测试,它基于现有的指令数据集构建。具体而言,在我们的LongIns中,我们引入了三种评估设置:全局指令与单任务(GIST)、局部指令与单任务(LIST)以及局部指令与多任务(LIMT)。基于LongIns,我们对现有的LLMs进行了全面评估,并得出以下重要发现:(1)表现最佳的GPT-4在128k上下文长度下,在我们的LongIns中16k的评估上下文窗口上表现不佳。(2)对于许多现有LLMs的多跳推理能力,在短上下文窗口(小于4k)下仍需付出巨大努力。

英文摘要:

The long-context capabilities of large language models (LLMs) have been a hot topic in recent years. To evaluate the performance of LLMs in different scenarios, various assessment benchmarks have emerged. However, as most of these benchmarks focus on identifying key information to answer questions, which mainly requires the retrieval ability of LLMs, these benchmarks can partially represent the reasoning performance of LLMs from large amounts of information. Meanwhile, although LLMs often claim to have context windows of 32k, 128k, 200k, or even longer, these benchmarks fail to reveal the actual supported length of these LLMs. To address these issues, we propose the LongIns benchmark dataset, a challenging long-context instruction-based exam for LLMs, which is built based on the existing instruction datasets. Specifically, in our LongIns, we introduce three evaluation settings: Global Instruction & Single Task (GIST), Local Instruction & Single Task (LIST), and Local Instruction & Multiple Tasks (LIMT). Based on LongIns, we perform comprehensive evaluations on existing LLMs and have the following important findings: (1). The top-performing GPT-4 with 128k context length performs poorly on the evaluation context window of 16k in our LongIns. (2). For the multi-hop reasoning ability of many existing LLMs, significant efforts are still needed under short context windows (less than 4k).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑