LiveOIBench:大语言模型在信息学奥林匹克竞赛中能超越人类参赛者吗?
LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?
AI总结:
研究大语言模型在信息学奥林匹克竞赛中的表现,引入含403个专家策划问题的LiveOIBench基准,通过四个关键特性评估34个模型,发现GPT - 5等不及顶级人类参赛者,还表明强大推理模型特点及基准数据污染少。
摘要:
由于竞赛编程问题的复杂性和易于验证,它们越来越多地被用于评估大语言模型(LLMs)的编码能力。然而,当前的编码基准存在局限性。为解决这些问题,我们引入了LiveOIBench,这是一个大规模竞赛编程基准,包含403个专家策划的问题,每个问题平均有60个官方测试用例。LiveOIBench有四个关键特性。对34个流行的通用和推理LLMs进行基准测试发现,GPT - 5达到第81.76百分位,仍不及顶级人类参赛者,开放权重模型中GPT - OSS - 120B仅达到第60百分位。推理跟踪分析表明强大的推理模型优先进行精确的问题分析而非过度探索。最后,跨发布日期、任务熟悉度和代码相似度分析发现我们的基准中数据污染证据极少。
英文摘要:
Competitive programming problems are increasingly used to evaluate the coding capabilities of large language models (LLMs) due to their complexity and ease of verification. Yet, current coding benchmarks face limitations such as a lack of exceptionally challenging problems, insufficient test case coverage, and reliance on online platform APIs that limit accessibility. To address these issues, we introduce LiveOIBench, a large-scale competitive programming benchmark featuring 403 expert-curated problems, averaging 60 official test cases each, drawn from 72 contests across 14 Informatics Olympiads held between 2023 and 2025. LiveOIBench has four key features: (1) expert-designed tasks with detailed subtask rubrics and extensive test cases; (2) direct comparison to elite human contestants; (3) continuous updates to reduce contamination risk; and (4) a fully offline, reproducible evaluation system. Benchmarking 34 popular general-purpose and reasoning LLMs, we find that GPT-5 achieves an 81.76th percentile, still falling short of top human contestants, while among the open-weight models, GPT-OSS-120B reaches only the 60th percentile. Reasoning-trace analyses indicate that robust reasoning models prioritize precise problem analysis over excessive exploration. Finally, analyses across release dates, task familiarity, and code similarity find minimal evidence of data contamination in our benchmark. Our leaderboard, code, and data are available at: https://liveoibench.github.io/.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。