arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.15089cs.CL
收藏

LongReason:一个通过上下文扩展构建的合成长上下文推理基准

LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Zhan Ling, Kang Liu, Kai Yan, Yifan Yang, Weijian Lin, Ting-Han Fan, Lingfeng Shen, Zhengyin Du, Jiecao Chen

AI总结:

针对现有长上下文推理基准覆盖不足的问题,本文提出通过上下文扩展构建的合成基准LongReason,对21个大语言模型评估后发现多数模型随上下文变长性能显著下降,该基准已开源。

摘要:

大语言模型(LLMs)在理解长上下文输入方面已取得显著进展,但用于评估LLMs长上下文推理能力的基准发展滞后。现有基准往往聚焦于范围较窄的任务,或是不需要复杂推理的任务。为填补这一空白、实现对当前LLMs长上下文推理能力的更全面评估,我们提出了全新的合成基准LongReason,该基准通过上下文扩展技术,从多样化的短上下文推理问题集合中生成长上下文推理问题。LongReason包含794道选择题型推理题,覆盖阅读理解、逻辑推理、数学应用题三类任务,具备多样的推理模式。我们在LongReason上评估了21个LLMs,结果显示随着上下文长度增加,大多数模型的性能出现显著下降。进一步分析表明,即使是当前最先进的LLMs,在不同任务上实现稳健推理仍有很大提升空间。我们已将LongReason开源至https://huggingface.co/datasets/lz1bytedance/LongReason,以支持对LLMs长上下文推理能力的全面评估。

英文摘要:

Large language models (LLMs) have demonstrated remarkable progress in understanding long-context inputs. However, benchmarks for evaluating the long-context reasoning abilities of LLMs fall behind the pace. Existing benchmarks often focus on a narrow range of tasks or those that do not demand complex reasoning. To address this gap and enable a more comprehensive evaluation of the long-context reasoning capabilities of current LLMs, we propose a new synthetic benchmark, LongReason, which is constructed by synthesizing long-context reasoning questions from a varied set of short-context reasoning questions through context expansion. LongReason consists of 794 multiple-choice reasoning questions with diverse reasoning patterns across three task categories: reading comprehension, logical inference, and mathematical word problems. We evaluate 21 LLMs on LongReason, revealing that most models experience significant performance drops as context length increases. Our further analysis shows that even state-of-the-art LLMs still have significant room for improvement in providing robust reasoning across different tasks. We have open-sourced LongReason under https://huggingface.co/datasets/lz1bytedance/LongReason to support the comprehensive evaluation of LLMs' long-context reasoning capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑