LongProc:在长程序生成上对长上下文语言模型进行基准测试
LongProc: Benchmarking Long-Context Language Models on Long Procedural Generation
AI总结:
本文提出 LongProc 基准,通过六类长程序生成任务评估 23 个长上下文语言模型,发现现有模型在 2K–8K token 结构化长输出与长程连贯性上存在明显不足。
摘要:
现有的评估长上下文语言模型(LCLM)的基准主要聚焦于长上下文回忆,要求模型在处理数千个无关 token 的同时,依据少量关键片段生成简短回答。我们提出 LongProc(Long Procedural Generation,长程序生成),这是一个同时要求整合高度分散信息并进行长文本生成的新基准。LongProc 包含六类不同的程序生成任务,例如从 HTML 页面中提取结构化信息并转换为 TSV 格式,以及执行复杂的搜索流程以制定旅行计划。这些任务通过测试 LCLM 遵循详细程序指令、综合并推理分散信息,以及生成结构化长文本输出(最高可达 8K token)的能力,对其构成挑战。此外,由于这些任务遵循确定性流程并产生结构化输出,因此能够实现可靠的基于规则的评估。我们在 LongProc 上以三个难度级别评估了 23 个 LCLM,包括指令微调模型和近期的推理模型,最大输出 token 数分别设置为 500、2K 和 8K。值得注意的是,尽管所有受测模型都声称上下文窗口大小超过 32K token,但开放权重模型通常在 2K token 任务上表现不佳,而 GPT-4o 等闭源模型在 8K token 任务上出现显著性能下降。得益于长 CoT 训练,推理模型在长文本生成中取得了更强的整体表现。进一步分析表明,LCLM 难以在长文本生成中维持长程连贯性。这些发现揭示了当前 LCLM 的关键局限,并表明其仍有巨大的改进空间。数据和代码可在 https://princeton-pli.github.io/LongProc 获取。
英文摘要:
Existing benchmarks for evaluating long-context language models (LCLMs) primarily focus on long-context recall, requiring models to produce short responses based on a few critical snippets while processing thousands of irrelevant tokens. We introduce LongProc (Long Procedural Generation), a new benchmark that requires both the integration of highly dispersed information and long-form generation. LongProc consists of six diverse procedural generation tasks, such as extracting structured information from HTML pages into a TSV format and executing complex search procedures to create travel plans. These tasks challenge LCLMs by testing their ability to follow detailed procedural instructions, synthesize and reason over dispersed information, and generate structured, long-form outputs (up to 8K tokens). Furthermore, as these tasks adhere to deterministic procedures and yield structured outputs, they enable reliable rule-based evaluation. We evaluated 23 LCLMs, including instruction-tuned models and recent reasoning models, on LongProc at three difficulty levels, with the maximum number of output tokens set at 500, 2K, and 8K. Notably, while all tested models claim a context window size above 32K tokens, open-weight models typically falter on 2K-token tasks, and closed-source models like GPT-4o show significant degradation on 8K-token tasks. Reasoning models achieve stronger overall performance in long-form generation, benefiting from long CoT training. Further analysis reveals that LCLMs struggle to maintain long-range coherence in long-form generations. These findings highlight critical limitations in current LCLMs and suggest substantial room for improvement. Data and code available at: https://princeton-pli.github.io/LongProc.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。