With reasoning language models such as OpenAI-o3 and DeepSeek-R1 emerging, large language models (LLMs) have entered a new phase of development. However, existing benchmarks for coding evaluation are gradually inadequate to assess the capability of advanced LLMs in code reasoning. To bridge the gap for high-level code reasoning assessment, we propose ProBench to benchmark LLMs in competitive programming, drawing inspiration from the International Collegiate Programming Contest. ProBench collects a comprehensive set of competitive programming problems from Codeforces, Luogu, and Nowcoder platforms during the period from July to December 2024, obtaining real test results through online submissions to ensure the fairness and accuracy of the evaluation. We establish a unified problem attribute system, including difficulty grading and algorithm tagging. With carefully collected and annotated data in ProBench, we systematically assess 9 latest LLMs in competitive programming across multiple dimensions, including thought chain analysis, error type diagnosis, and reasoning depth evaluation. Experimental results show that QwQ-32B-Preview achieves the best score of 20.93 followed by DeepSeek-V3 with a score of 16.38, suggesting that models trained with specialized reasoning tasks significantly outperform general-purpose models (even larger than reasoning-oriented models) in programming. Further analysis also reveals key areas for programming capability enhancement, e.g., algorithm adaptability and reasoning sufficiency, providing important insights for the future development of reasoning models.
InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information
School of Cyber Science and Engineering, Huazhong University of Science and Technology(华中科技大学网络空间科学与工程学院);Key Laboratory of Cyberspace Security, Ministry of Education(教育部网络空间安全重点实验室);Hubei Key Laboratory of Distributed System Security(湖北省分布式系统安全重点实验室)
UBench: Benchmarking Uncertainty in Large Language Models with Multiple Choice Questions
College of Software, Nankai University(南开大学软件学院);Institute for Advanced Algorithms Research (Shanghai)(上海先进算法研究所);College of Artificial Intelligence, Tianjin University of Science and Technology(天津科技大学人工智能学院);Tianjin Key Laboratory of Software Experience and Human Computer Interaction(天津软件体验与人机交互重点实验室)
Benchmarking LLM Judges for Mobile Agent Evaluation
Mila – Québec AI Institute(米拉-魁北克人工智能研究所);Concordia University(康考迪亚大学);University of Toronto(多伦多大学);Shanghai University(上海大学);McMaster University(麦克马斯特大学)
Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problems
Institute for Infocomm Research (I 2 R), A*STAR, Singapore(信息与通信研究 institute(I2R),A*STAR,新加坡);Centre for Frontier AI Research (CFAR), A*STAR, Singapore(前沿人工智能研究 centre(CFAR),A*STAR,新加坡)
RefactorCoderQA: Benchmarking LLMs for Multi-Domain Coding Question Solutions in Cloud and Edge Deployment
York University and Algoma University(约克大学和阿尔戈马大学);Algoma University(阿尔戈马大学);Department of Systems and Computer Engineering, Carleton University(系统与计算机工程系,卡尔顿大学);Department of Math and Computer Science, Brandon University(数学与计算机科学系,布兰登大学)
FinBoardBench: Benchmarking Dynamic Wealth Management and Strategic Financial Reasoning of LLMs via Board Game Simulations
School of Computer Science and Engineering, Macau University of Science and Technology, Macau, China(1 计算机科学与工程学院,澳门科技大学,澳门,中国);School of Economics, Anhui University, Anhui, China(2 经济学院,安徽大学,安徽,中国);SKLPlanets, Macau University of Science and Technology, Macau, China(3 SKLPlanets,澳门科技大学,澳门,中国);Department of Computer and Information Science, University of Macau, Macau, China(4 计算机与信息科学系,澳门大学,澳门,中国);School of Energy and Power Engineering, Huazhong University of Science and Technology, Hubei, China(5 能源与动力工程学院,华中科技大学,湖北,中国)
EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective
Hong Kong University of Science and Technology (Guangzhou)(香港理工大学(广州));Createlink Technology(创-link科技);Beijing University of Posts and Telecommunications(北京邮电大学);Beijing Institute of Technology(北京理工大学)