arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.02060cs.CLcs.AI
收藏

长上下文 LLM 在长上下文内学习中表现不佳

Long-context LLMs Struggle with Long In-context Learning

University of Waterloo(滑铁卢大学);Carnegie Mellon University(卡内基梅隆大学);Vector Institute(向量研究所)

Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, Wenhu Chen

AI总结:

本文提出 LongICLBench,通过 28–174 类、2K–50K token 的极端标签分类任务评估 15 个长上下文 LLM,揭示其在长上下文理解、信息整合推理和标签位置偏置上的不足。

摘要:

大语言模型(LLM)在处理长序列方面已取得显著进展。Gemini 等部分模型甚至似乎能够处理数百万个 token。然而,对其性能的评估在很大程度上仍局限于困惑度和合成任务等指标,这些指标可能无法充分反映它们在更具挑战性的真实场景中的真实能力。我们引入了一个用于极端标签分类中长上下文内学习的基准(LongICLBench),该基准使用六个数据集,类别数为 28 至 174,输入长度为 2K 至 50K token。我们的基准要求 LLM 理解整个输入,以识别庞大的标签空间并作出正确预测。我们对 15 个长上下文 LLM 进行了评估,发现它们在标签空间较小且演示较短、挑战性较低的分类任务上表现良好。然而,它们在 Discovery 等具有 174 个标签、更具挑战性的任务上表现不佳,表明其在处理较长且上下文丰富的序列方面存在能力差距。进一步分析揭示,模型偏向序列中稍后出现的标签,并且需要改进对多条信息的推理能力。我们的研究表明,长上下文理解和推理对现有 LLM 而言仍是一项具有挑战性的任务。我们相信 LongICLBench 可以作为未来长上下文 LLM 更贴近现实的评估。

英文摘要:

Large Language Models (LLMs) have made significant strides in handling long sequences. Some models like Gemini could even to be capable of dealing with millions of tokens. However, their performance evaluation has largely been confined to metrics like perplexity and synthetic tasks, which may not fully capture their true abilities in more challenging, real-world scenarios. We introduce a benchmark (LongICLBench) for long in-context learning in extreme-label classification using six datasets with 28 to 174 classes and input lengths from 2K to 50K tokens. Our benchmark requires LLMs to comprehend the entire input to recognize the massive label spaces to make correct predictions. We evaluate on 15 long-context LLMs and find that they perform well on less challenging classification tasks with smaller label space and shorter demonstrations. However, they struggle with more challenging task like Discovery with 174 labels, suggesting a gap in their ability to process long, context-rich sequences. Further analysis reveals a bias towards labels presented later in the sequence and a need for improved reasoning over multiple pieces of information. Our study reveals that long context understanding and reasoning is still a challenging task for the existing LLMs. We believe LongICLBench could serve as a more realistic evaluation for the future long-context LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑