arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.20868cs.CL
收藏

ProBench:大型语言模型在竞争性编程中的基准测试

ProBench: Benchmarking Large Language Models in Competitive Programming

Tianjin University(天津大学)

Lei Yang, Renren Jin, Ling Shi, Jianxiang Peng, Yue Chen, Deyi Xiong

AI总结:

提出ProBench基准,基于Codeforces等平台数据评估9个LLM的竞争性编程能力,发现专用推理模型优于通用模型,并指出算法适应性与推理充分性是关键提升方向。

摘要:

随着OpenAI-o3和DeepSeek-R1等推理语言模型的出现,大型语言模型(LLMs)进入了新的发展阶段。然而,现有的编码评估基准逐渐不足以评估先进LLM在代码推理方面的能力。为了弥合高级代码推理评估的差距,我们受国际大学生程序设计竞赛启发,提出了ProBench来对LLM进行竞争性编程基准测试。ProBench从Codeforces、Luogu和Nowcoder平台收集了2024年7月至12月期间的一系列综合性竞争性编程问题,并通过在线提交获得真实测试结果,以确保评估的公平性和准确性。我们建立了一个统一的问题属性系统,包括难度分级和算法标注。利用ProBench中精心收集和标注的数据,我们从多个维度系统评估了9个最新的LLM在竞争性编程中的表现,包括思维链分析、错误类型诊断和推理深度评估。实验结果表明,QwQ-32B-Preview取得了最佳成绩20.93,其次是DeepSeek-V3,得分为16.38,这表明经过专门推理任务训练的模型在编程方面显著优于通用模型(甚至优于面向推理的更大模型)。进一步的分析还揭示了编程能力提升的关键领域,例如算法适应性和推理充分性,为推理模型的未来发展提供了重要见解。

英文摘要:

With reasoning language models such as OpenAI-o3 and DeepSeek-R1 emerging, large language models (LLMs) have entered a new phase of development. However, existing benchmarks for coding evaluation are gradually inadequate to assess the capability of advanced LLMs in code reasoning. To bridge the gap for high-level code reasoning assessment, we propose ProBench to benchmark LLMs in competitive programming, drawing inspiration from the International Collegiate Programming Contest. ProBench collects a comprehensive set of competitive programming problems from Codeforces, Luogu, and Nowcoder platforms during the period from July to December 2024, obtaining real test results through online submissions to ensure the fairness and accuracy of the evaluation. We establish a unified problem attribute system, including difficulty grading and algorithm tagging. With carefully collected and annotated data in ProBench, we systematically assess 9 latest LLMs in competitive programming across multiple dimensions, including thought chain analysis, error type diagnosis, and reasoning depth evaluation. Experimental results show that QwQ-32B-Preview achieves the best score of 20.93 followed by DeepSeek-V3 with a score of 16.38, suggesting that models trained with specialized reasoning tasks significantly outperform general-purpose models (even larger than reasoning-oriented models) in programming. Further analysis also reveals key areas for programming capability enhancement, e.g., algorithm adaptability and reasoning sufficiency, providing important insights for the future development of reasoning models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑