大模型推理能力

AnomalyGen: An Automated Semantic Log Sequence Generation Framework with LLM for Anomaly Detection

Xinyu Li, Yingtong Huo, Chenxi Mao, Shiwen Shan, Yuxin Su, Dan Li, Zibin Zheng

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；CoT(abstract)

LLMs Are Not Intelligent Thinkers: Introducing Mathematical Topic Tree Benchmark for Comprehensive Evaluation of LLMs

Arash Gholami Davoodi, Seyed Pouyan Mousavi Davoudi, Pouya Pezeshkpour

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI、cs.LG

DeepSound-V1: Start to Think Step-by-Step in the Audio Generation from Videos

Yunming Liang, Zihao Chen, Chaofan Ding, Xinhan Di

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；CoT(abstract)

Comments 11 pages, 6 figures

Benchmarking Agentic Workflow Generation

Shuofei Qiao, Runnan Fang, Zhisong Qiu, Xiaobin Wang, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, Huajun Chen

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICLR 2025

RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiques

Zhengyang Tang, Ziniu Li, Zhenyang Xiao, Tian Ding, Ruoyu Sun, Benyou Wang, Dayiheng Liu, Fei Huang, Tianyu Liu, Bowen Yu, Junyang Lin

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI、cs.LG

Episodic Memories Generation and Evaluation Benchmark for Large Language Models

Alexis Huet, Zied Ben Houidi, Dario Rossi

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI、cs.LG

QuantumLLMInstruct: A 500k LLM Instruction-Tuning Dataset with Problem-Solution Pairs for Quantum Computing

Shlomo Kashani

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；CoT(abstract)

VisGraphVar: A Benchmark Generator for Assessing Variability in Graph Analysis Using Large Vision-Language Models

Camilo Chacón Sartori, Christian Blum, Filippo Bistaffa

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI、cs.LG

Towards More Effective Table-to-Text Generation: Assessing In-Context Learning and Self-Evaluation with Open-Source Models

Sahar Iravani, Tim . O . F Conrad

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 15 pages

Investigating the Transferability of Code Repair for Low-Resource Programming Languages

Kyle Wong, Alfonso Amayuelas, Liangming Pan, William Yang Wang

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI、cs.LG

CELLO: Causal Evaluation of Large Vision-Language Models

Meiqi Chen, Bo Peng, Yan Zhang, Chaochao Lu

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；CoT(abstract)

Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves

Yihe Deng, Weitong Zhang, Zixiang Chen, Quanquan Gu

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 25 pages, 7 figures, 22 tables

CogBench: a large language model walks into a psychology lab

Julian Coda-Forno, Marcel Binz, Jane X. Wang, Eric Schulz

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI、cs.LG

SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Models

Manish Nagireddy, Lamogha Chiazor, Moninder Singh, Ioana Baldini

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments AAAI 2024

Agents meet OKR: An Object and Key Results Driven Agent System with Hierarchical Self-Collaboration and Self-Evaluation

Yi Zheng, Chongyang Ma, Kanle Shi, Haibin Huang

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；self-correction(abstract)

Prompted LLMs as Chatbot Modules for Long Open-domain Conversation

Gibbeum Lee, Volker Hartmann, Jongho Park, Dimitris Papailiopoulos, Kangwook Lee

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted to the Findings of ACL2023. The camera-ready version with additional experimental results will be uploaded

An Evaluation Dataset for Legal Word Embedding: A Case Study On Chinese Codex

Chun-Hsien Lin, Pu-Jen Cheng

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 16 pages, 9 figures, 3rd International Conference on Natural Language Computing and AI (NLCAI 2022)

MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedback

Zonghai Yao, Aditya Parashar, Huixue Zhou, Won Seok Jang, Feiyun Ouyang, Zhichao Yang, Hong Yu

专题命中 推理评测 ：self-correction(abstract,comments)；reasoning(abstract)；分类 cs.CL、cs.AI

Comments Equal contribution for the first two authors. To appear in proceedings of the Main Conference on 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL). Keywords: Question Generation, USMLE, Self-Refine, Self-Critique, and Self-Correction, LLM-as-Judge, AI for Medical Education

ImmigrationReason: A Structured Dataset of U.S. Immigration Appeals for Legal Reasoning Research

ImmigrationReason：面向法律推理研究的美国移民上诉结构化数据集

Amirhossein Afsharrad, Seyed Shahabeddin Mousavi

机构 * Stanford University（斯坦福大学） 纠错

专题命中 推理评测 ：reasoning(title)；分类 cs.CL

AI总结 本研究推出ImmigrationReason大规模结构化数据集，涵盖美国移民上诉裁决相关数据，可支撑法律推理领域的结果预测、错误分析及高风险监管智能体设计等研究方向。

AMPLIFAI: A Multiphase CT Dataset for Benchmarking Clinical Reasoning in LI-RADS Assessment of Liver Lesions

AMPLIFAI：用于基准测试LI-RADS肝脏病变评估临床推理的多期CT数据集

Pranav Kulkarni, Nikhil Shah, Amritansh Suryavanshi, Jana G. Delfino, James Tonascia, Jade Wong-You-Cheong, Barton Lane, Joseph Chirico, Jeffrey D. Hirsch, Ang Li, Heng Huang, Florence X. Doo

机构 * University of Maryland, College Park（马里兰大学帕克分校） ； University of Maryland School of Medicine（马里兰大学医学院） ； University of Maryland Institute for Health Computing（马里兰大学健康计算研究所） ； University of Maryland Medical System（马里兰大学医学系统） 纠错

专题命中 推理评测 ：reasoning(title)；分类 cs.LG

AI总结 研究针对LI-RADS肝脏病变评估AI模型缺乏公开高质量数据集的问题，推出首个公开多期腹部CT扫描的AMPLIFAI数据集，标注LI-RADS类别并分割关键特征，助力相关透明可复现研究。

Comments 15 pages, 6 figures, 3 tables

Large Language Models Show Metacognitive Sensitivity in Medical Reasoning

大型语言模型在医学推理中表现出元认知敏感性

Ahmad Nazzal

专题命中 推理评测 ：reasoning(title)；分类 cs.AI

AI总结 本研究开发临床基准测试医学LLM，发现其在AT-NCD与DRCI鉴别中具部分元认知敏感性，错误集中于冲突病例，建立了医学LLM相关评估框架。

Beyond Local Accuracy: A Protocol-Level Identifiability Audit for Controlled LLM Reasoning Evaluation

超越局部准确率：面向受控大语言模型推理评估的协议级可识别性审计

Junhao Luo, Ning Huang, Ziqi Sha, Wenxuan Tang, Wei Deng

机构 * School of Statistics and Data Science, Southwestern University of Finance and Economics（西南财经大学统计与数据科学学院） 纠错

专题命中 推理评测 ：reasoning(title)；分类 cs.CL

AI总结 该研究提出协议级可识别性审计方法，无需调用模型即可验证LLM评估协议的有效性，发现基础准确率与选择性响应保真度存在显著差异，还确定了冻结策略类的最小识别支持。

Comments 15 pages, 9 figures. Ning Huang, Ziqi Sha, and Wenxuan Tang contributed equally as second authors. Wei Deng is the corresponding author

mChartQA: A universal benchmark for multimodal Chart Question Answer based on Vision-Language Alignment and Reasoning

Jingxuan Wei, Nan Xu, Guiyong Chang, Yin Luo, BiHui Yu, Ruifeng Guo

专题命中 推理评测 ：reasoning(title)；分类 cs.AI

Journal ref Pattern Recognition 172 (2026) 112348

SP-Mind: An Autonomous Reasoning Agent for Spatial Proteomics Analysis

SP-Mind: 用于空间蛋白质组学分析的自主推理智能体

Yucheng Yuan, Yuanfeng Ji, Zhongxiao Li, Ruijiang Li

机构 * Department of Computer Science, Stanford University, Stanford, USA（计算机科学系，斯坦福大学，斯坦福，美国） ； Department of Radiation Oncology, Stanford University, Stanford, USA（放射肿瘤学系，斯坦福大学，斯坦福，美国） 纠错

专题命中 推理评测 ：reasoning(title)；分类 cs.AI

AI总结 提出首个自主AI智能体SP-Mind，统一空间蛋白质组学分析流程，通过自然语言查询实现端到端分析，在SP-Bench基准上达到最优性能。

Comments 24 pages, 6 figures. Accepted to ICML 2026. Equal contribution by Yucheng Yuan and Yuanfeng Ji

Zero-shot reasoning for simulating scholarly peer-review

模拟学术同行评审的零样本推理

Khalid M. Saqr

机构 * College of Engineering and Technology（工程与技术学院） ； Arab Academy for Science, Technology, and Maritime Transport（阿拉伯科学、技术与海运学院） 纠错

专题命中 推理评测 ：reasoning(title)；分类 cs.AI

AI总结 本研究构建同行评审模拟基准 xPeer，对比其与人类评审的特征差异，发现 xPeer 报告针对性等更优，人类报告显式推理等更强，成果存档于 Zenodo。

Taxonomy-Targeted Error Generation for Quantitative Reasoning

错误作为透镜：通过合成误解生成探究LLM推理

Xinming Yang, Jun Li

机构 * CUNY Graduate Center（纽约大学研究生中心） ； CUNY Queens College（纽约市立大学皇后学院） 纠错

专题命中 推理评测 ：reasoning(title)；分类 cs.CL

AI总结 提出一个框架，通过生成针对Bloom分类学五类错误的合成误解，以诊断LLM推理能力，并发现目标错误生成比自由形式错误生成更难。

Beyond Correctness: Enhancing Architectural Reasoning in Code LLMs via Scalable Labeling with Agentic Judgment

超越正确性：通过可扩展的智能体判断标注增强代码大模型的架构推理能力

Kirill Vasilevski, Ximing Dong, Benjamin Rombaut, Milad Soltany, Ruochen Deng, Jiahuei Lin, Arthur Leung, Dayi Lin, Boyuan Chen, Shaowei Wang, Ahmed E. Hassan

机构 * Centre for Software Excellence, Huawei Canada（华为加拿大软件卓越中心） ； Department of Computer Science, University of Manitoba, Canada（曼尼托巴大学计算机科学系） ； School of Computing, Queen’s University, Canada（皇后大学计算科学学院） 纠错

专题命中 推理评测 ：reasoning(title)；分类 cs.AI

AI总结 针对代码大模型缺乏架构理解的问题，提出智能体判断流水线，利用强LLM作为专家架构评估的代理，通过两个判断器（ACJ和AQJ）实现可扩展标注，微调模型在SWE-bench上提升高达540%，并展现跨语言泛化能力。

Position: Vision-Language-Action Models Cannot Be Verified to Perform Physical Reasoning

立场：视觉-语言-动作模型无法被验证执行物理推理

Taozhao Chen, Ian Manchester, Huaming Chen

专题命中 推理评测 ：reasoning(title)；分类 cs.AI

AI总结 本文指出基于预训练视觉-语言模型的VLA系统在机器人操作基准上的性能提升无法区分语义匹配与物理泛化，现有评估指标不能验证物理推理能力。

Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models

MLLM 能否像人类一样进行批评？评估多模态大语言模型中的开放式审美推理

Sajjad Ghiasvand, Maryam Amirizaniani, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh, Ramtin Pedarsani

机构 * UCSB（加州大学圣塔芭芭拉分校） ； University of Washington（华盛顿大学） ； UCLA（加州大学洛杉矶分校） 纠错

专题命中 推理评测 ：reasoning(title)；分类 cs.CL

AI总结 本文评估多模态大语言模型在开放式审美批评中的表现，发现基于参考的相似度指标存在误导，模型在选择性、特异性和多样性方面与人类批评存在系统性差异。

Frame-Conditioned Moral Computation in LLaMA 3.1-8B-Instruct: A Mechanistic Interpretability Audit of Ethical Reasoning

LLaMA 3.1-8B-Instruct中的框架条件化道德计算：伦理推理的机械可解释性审计

Ali Dasdan, Manan Shah, W. Russell Neuman, Chad Coleman, Kund Meghani, Safinah Ali

机构 * KD Consulting, CA, USA（KD咨询公司，美国加利福尼亚州） ； New York University, NY, USA（纽约大学，美国纽约州） 纠错

专题命中 推理评测 ：reasoning(title)；分类 cs.AI

AI总结 通过机械可解释性平台分析LLaMA 3.1-8B-Instruct在54个道德提示上的内部计算，发现情境锚定效应：领域特定表示主导激活列表顶部，模型道德能力恒定但显著性高度依赖于提示选择的解释框架。

Comments 47 pages, 10 figures