RiddleBench: A New Generative Reasoning Benchmark for LLMs
Nilekani Centre at AI4Bharat(AI4Bharat的Nilekani中心);Indian Institute of Technology Madras(印度理工学院马德拉斯分校);IT University of Copenhagen(哥本哈根技术大学);Microsoft(微软);Indian Institute of Engineering, Science and Technology, Shibpur(Shibpur印度工程科学技术学院)
Large Language Models have demonstrated strong performance on many established reasoning benchmarks. However, these benchmarks primarily evaluate structured skills like quantitative problem-solving, leaving a gap in assessing flexible, multifaceted reasoning abilities that are central to human intelligence. These abilities require integrating logical deduction with spatial awareness and constraint satisfaction, which current evaluations do not measure well. To address this, we introduce RiddleBench, a benchmark of 1,737 challenging puzzles in English designed to probe these core reasoning capabilities. Evaluation of state-of-the-art models on RiddleBench shows fundamental weaknesses. Even top proprietary models like Gemini 2.5 Pro, o3, and Claude 4 Sonnet achieve accuracy just above 60% (60.30%, 63.37%, and 63.16%). Analysis further reveals deep failures, including hallucination cascades (accepting flawed reasoning from other models) and poor self-correction due to a strong self-confirmation bias. Their reasoning is also fragile, with performance degrading significantly when constraints are reordered or irrelevant information is introduced. RiddleBench functions as a diagnostic tool for these issues and as a resource for guiding the development of more robust and reliable language models.
Overview of PAN 2026: Voight-Kampff Generative AI Detection, Text Watermarking, Multi-Author Writing Style Analysis, Generative Plagiarism Detection, and Reasoning Trajectory Detection
Friedrich Schiller University Jena(耶纳弗里德里希·施勒格尔大学);University of Applied Sciences BFI(应用科学大学BFI);Mohamed bin Zayed University of Artificial Intelligence(马尔代夫 bin Zayed 人工智能大学);University of Kassel(卡塞尔大学);INSAIT, Sofia University ``St. Kliment Ohridski''(INSAIT,索菲亚大学『圣克莱门特·奥赫里迪斯』);University of Innsbruck(因斯布鲁克大学)
PAN 2026旨在通过客观评估推进计算风格学和文本取证,涵盖生成式AI检测、文本水印、多作者写作分析、抄袭检测及推理轨迹检测等五个新旧任务。
Generative Data Augmentation for Commonsense Reasoning
Northwestern University(西北大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Pennsylvania(宾夕法尼亚大学);Carnegie Mellon University(卡内基梅隆大学);University of Washington(华盛顿大学)
GRAM-R$^2$: Self-Training Generative Foundation Reward Models for Reward Reasoning
School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);Pattern Recognition Center, WeChat AI, Tencent Inc.(腾讯公司微信AI模式识别中心)
Generative Visual Commonsense Answering and Explaining with Generative Scene Graph Constructing
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics(南京航空航天大学人工智能学院);The Hong Kong Polytechnic University(香港理工大学);The Chinese University of Hong Kong(香港中文大学)
Generative Evaluation of Complex Reasoning in Large Language Models
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Institute for AI Industry Research, Tsinghua University(清华大学人工智能产业研究院);Computer Science Department, Stanford University(斯坦福大学计算机科学系)
Combinatorial Reasoning: Selecting Reasons in Generative AI Pipelines via Combinatorial Optimization
Icosa Computing Inc.(伊科萨计算公司);NASA ARC(美国国家航空航天局艾姆斯研究中心);USRA Research Institute for Advanced Computer Science (RIACS)(美国大学空间研究协会高级计算机科学研究所);Hewlett Packard Labs(惠普实验室);DataStax(DataStax公司)
本文提出完全自动化的组合推理框架 CR,将 LLM 流水线采样的理由建模为 QUBO 问题并选择子集构建思维链提示,实验表明组合优化可增强生成式 AI 推理。
Contextualized Scene Imagination for Generative Commonsense Reasoning
University of Southern California(南加州大学);Information Sciences Institute, University of Southern California(南加州大学信息科学研究所);University of California, San Diego(加州大学圣地亚哥分校)
MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation
University of Wisconsin–Madison(威斯康星大学麦迪逊分校);University of California, Los Angeles(加州大学洛杉矶分校);Michigan State University(密歇根州立大学);University of Illinois Urbana–Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Adelaide(阿德莱德大学);Salesforce AI Research(Salesforce人工智能研究);Microsoft(微软);Adobe Research(Adobe研究)
TRIGO: Benchmarking Formal Mathematical Proof Reduction for Generative Language Models
Sun Yat-Sen University(中山大学);Peking University(北京大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);City University of Hong Kong(香港城市大学);The Chinese University of Hong Kong(香港中文大学)
KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning
University of Illinois at Chicago(伊利诺伊大学芝加哥分校);Huazhong University of Science and Technology(华中科技大学);Lehigh University(里海大学);Beihang University(北京航空航天大学)