LongCodeBench：在100万上下文窗口下评估编码大语言模型
LongCodeBench: Evaluating Coding LLMs at 1M Context Windows
AI总结：
本研究推出长上下文编码能力基准LongCodeBench，基于真实GitHub问题构建问答与bug修复任务，分层评估不同规模模型，发现长上下文仍是各类模型的短板，性能出现显著下滑。
摘要：
模型的上下文长度增长迅速，仅在短短几年内就从数千个token增长到数百万个。现代长上下文模型的极端上下文规模使得构建真实的长上下文基准变得困难——不仅因为收集百万上下文任务的成本高昂，还因为难以找到需要大量上下文的真实场景。我们将代码理解与修复作为长上下文模型的天然测试平台和挑战性任务，并推出LongCodeBench（LCB，长代码基准），这是一个用于测试长上下文场景下大语言模型编码能力的基准。我们的基准通过取材于真实GitHub问题，构建问答（LongCodeQA，长代码问答）和错误修复（LongSWE-Bench，长软件工程师基准）任务，在真实且重要的场景中测试长上下文大语言模型的理解和修复能力。我们精心对基准的复杂度进行分层，使我们能够评估不同规模的模型——从Qwen2.5 14B Instruct到谷歌的旗舰Gemini模型。我们发现长上下文仍然是所有模型的弱点，性能出现下降，例如Claude 3.5 Sonnet的性能从29%降至3%，Qwen2.5的性能从70.2%降至40%。LCB数据集已在https://huggingface.co/datasets/Steefano/LCB公开提供，复现本文工作的代码库可在https://github.com/Zteefano/long-code-bench获取。
英文摘要：
Context lengths for models have grown rapidly, from thousands to millions of tokens in just a few years. The extreme context sizes of modern long-context models have made it difficult to construct realistic long-context benchmarks -- not only due to the cost of collecting million-context tasks but also in identifying realistic scenarios that require significant contexts. We identify code comprehension and repair as a natural testbed and challenge task for long-context models and introduce LongCodeBench (LCB), a benchmark to test LLM coding abilities in long-context scenarios. Our benchmark tests both the comprehension and repair capabilities of LCLMs in realistic and important settings by drawing from real-world GitHub issues and constructing QA (LongCodeQA) and bug fixing (LongSWE-Bench) tasks. We carefully stratify the complexity of our benchmark, enabling us to evaluate models across different scales -- ranging from Qwen2.5 14B Instruct to Google's flagship Gemini model. We find that long-context remains a weakness for all models, with performance drops such as from 29% to 3% for Claude 3.5 Sonnet, or from 70.2% to 40% for Qwen2.5. The LCB dataset is available publicly at https://huggingface.co/datasets/Steefano/LCB and the codebase to replicate the work on this paper at https://github.com/Zteefano/long-code-bench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
