Generative Evaluation of Complex Reasoning in Large Language Models
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Institute for AI Industry Research, Tsinghua University(清华大学人工智能产业研究院);Computer Science Department, Stanford University(斯坦福大学计算机科学系)
Haowei Lin, Xiangyu Wang, Ruilin Yan, Baizhou Huang, Haotian Ye, Jianhua Zhu, Zihao Wang, James Zou, Jianzhu Ma, Yitao Liang
With powerful large language models (LLMs) demonstrating superhuman reasoning capabilities, a critical question arises: Do LLMs genuinely reason, or do they merely recall answers from their extensive, web-scraped training datasets? Publicly released benchmarks inevitably become contaminated once incorporated into subsequent LLM training sets, undermining their reliability as faithful assessments. To address this, we introduce KUMO, a generative evaluation framework designed specifically for assessing reasoning in LLMs. KUMO synergistically combines LLMs with symbolic engines to dynamically produce diverse, multi-turn reasoning tasks that are partially observable and adjustable in difficulty. Through an automated pipeline, KUMO continuously generates novel tasks across open-ended domains, compelling models to demonstrate genuine generalization rather than memorization. We evaluated 23 state-of-the-art LLMs on 5,000 tasks across 100 domains created by KUMO, benchmarking their reasoning abilities against university students. Our findings reveal that many LLMs have outperformed university-level performance on easy reasoning tasks, and reasoning-scaled LLMs reach university-level performance on complex reasoning challenges. Moreover, LLM performance on KUMO tasks correlates strongly with results on newly released real-world reasoning benchmarks, underscoring KUMO's value as a robust, enduring assessment tool for genuine LLM reasoning capabilities.
School of Informatics, Xiamen University, China(厦门大学信息学院);WeChat AI, Tencent Inc, China(腾讯公司微信AI部门);Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China(福建省和台湾非物质文化遗产数字化保护与智能处理重点实验室);Shanghai Artificial Intelligence Laboratory, China(上海人工智能实验室)
Parameterized Argumentation-based Reasoning Tasks for Benchmarking Generative Language Models
Bernoulli Institute of Mathematics, Computer Science and Artificial Intelligence, University of Groningen(伯努利数学、计算机科学和人工智能研究所,格罗宁根大学);Department of Information and Computing Sciences, Utrecht University(信息与计算科学系,乌得勒支大学)
Omni-R1: Towards the Unified Generative Paradigm for Multimodal Reasoning
The Hong Kong Polytechnic University(香港理工大学);Singapore Management University(新加坡国立大学);Shandong University(山东大学);University of Science and Technology of China(中国科学技术大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
GRAM-R$^2$: Self-Training Generative Foundation Reward Models for Reward Reasoning
School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);Pattern Recognition Center, WeChat AI, Tencent Inc.(腾讯公司微信AI模式识别中心)
Combinatorial Reasoning: Selecting Reasons in Generative AI Pipelines via Combinatorial Optimization
Icosa Computing Inc.(伊科萨计算公司);NASA ARC(美国国家航空航天局艾姆斯研究中心);USRA Research Institute for Advanced Computer Science (RIACS)(美国大学空间研究协会高级计算机科学研究所);Hewlett Packard Labs(惠普实验室);DataStax(DataStax公司)
本文提出完全自动化的组合推理框架 CR,将 LLM 流水线采样的理由建模为 QUBO 问题并选择子集构建思维链提示,实验表明组合优化可增强生成式 AI 推理。
Overview of PAN 2026: Voight-Kampff Generative AI Detection, Text Watermarking, Multi-Author Writing Style Analysis, Generative Plagiarism Detection, and Reasoning Trajectory Detection
Friedrich Schiller University Jena(耶纳弗里德里希·施勒格尔大学);University of Applied Sciences BFI(应用科学大学BFI);Mohamed bin Zayed University of Artificial Intelligence(马尔代夫 bin Zayed 人工智能大学);University of Kassel(卡塞尔大学);INSAIT, Sofia University ``St. Kliment Ohridski''(INSAIT,索菲亚大学『圣克莱门特·奥赫里迪斯』);University of Innsbruck(因斯布鲁克大学)
PAN 2026旨在通过客观评估推进计算风格学和文本取证,涵盖生成式AI检测、文本水印、多作者写作分析、抄袭检测及推理轨迹检测等五个新旧任务。
Generative Data Augmentation for Commonsense Reasoning
Northwestern University(西北大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Pennsylvania(宾夕法尼亚大学);Carnegie Mellon University(卡内基梅隆大学);University of Washington(华盛顿大学)
Generative Visual Commonsense Answering and Explaining with Generative Scene Graph Constructing
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics(南京航空航天大学人工智能学院);The Hong Kong Polytechnic University(香港理工大学);The Chinese University of Hong Kong(香港中文大学)
RiddleBench: A New Generative Reasoning Benchmark for LLMs
Nilekani Centre at AI4Bharat(AI4Bharat的Nilekani中心);Indian Institute of Technology Madras(印度理工学院马德拉斯分校);IT University of Copenhagen(哥本哈根技术大学);Microsoft(微软);Indian Institute of Engineering, Science and Technology, Shibpur(Shibpur印度工程科学技术学院)
Elucidating STEM Concepts through Generative AI: A Multi-modal Exploration of Analogical Reasoning
Carnegie Learning(卡内基学习公司);University of Sheffield(谢菲尔德大学);Microsoft Research(微软研究院);University of Maryland, College Park(马里兰大学帕克分校);University of Georgia(佐治亚大学);New York University(纽约大学);Carnegie Mellon University(卡内基梅隆大学)