大模型推理能力

Test-time Scaling Techniques in Theoretical Physics -- A Comparison of Methods on the TPBench Dataset

Zhiqi Gao, Tianyi Li, Yurii Kvasiuk, Sai Chaitanya Tadepalli, Maja Rudolph, Daniel J. H. Chung, Frederic Sala, Moritz Münchmeyer

机构 * Department of Physics, University of Wisconsin-Madison（威斯康星大学麦迪逊分校物理系） ； Department of Computer Science, University of Wisconsin-Madison（威斯康星大学麦迪逊分校计算机科学系） ； Data Science Institute (DSI), University of Wisconsin-Madison（威斯康星大学麦迪逊分校数据科学研究院） ； Department of Physics, Indiana University, Bloomington（印第安纳大学布卢明顿分校物理系） 纠错

专题命中 推理评测 ：reasoning(abstract)；verifier(abstract)；分类 cs.AI、cs.LG

Comments 23 pages, 6 figures

LOGICPO: Efficient Translation of NL-based Logical Problems to FOL using LLMs and Preference Optimization

Koushik Viswanadha, Deepanway Ghosal, Somak Aditya

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.AI、cs.LG

Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity

Hangfan Zhang, Zhiyao Cui, Jianhao Chen, Xinrun Wang, Qiaosheng Zhang, Zhen Wang, Dinghao Wu, Shuyue Hu

机构 * Pennsylvania State University（宾夕法尼亚州立大学） ； Northwestern Polytechnical University（西北工业大学） ； Singapore Management University（新加坡管理学院） ； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室） ； Nanjing University（南京大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.LG

Comments This position paper takes a critical view of the status quo of MAD research, and outline multiple potential directions to improve MAD

Efficient but Vulnerable: Benchmarking and Defending LLM Batch Prompting Attack

Murong Yue, Ziyu Yao

机构 * George Mason University（乔治·马歇尔大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.AI、cs.LG

Comments Accepted to ACL Findings, 2025

PsychBench: A comprehensive and professional benchmark for evaluating the performance of LLM-assisted psychiatric clinical practice

Shuyu Liu, Ruoxi Wang, Ling Zhang, Xuequan Zhu, Rui Yang, Xinzhu Zhou, Fei Wu, Zhi Yang, Cheng Jin, Gang Wang

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning

Zhong Zhang, Yaxi Lu, Yikun Fu, Yupeng Huo, Shenzhi Yang, Yesai Wu, Han Si, Xin Cong, Haotian Chen, Yankai Lin, Jie Xie, Wei Zhou, Wang Xu, Yuanheng Zhang, Zhou Su, Zhongwu Zhai, Xiaoming Liu, Yudong Mei, Jianming Xu, Hongyan Tian, Chongyi Wang, Chi Chen, Yuan Yao, Zhiyuan Liu, Maosong Sun

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI

Comments Updated results in Table 2 and Table 3; The project is available at https://github.com/OpenBMB/AgentCPM-GUI

Output Scaling: YingLong-Delayed Chain of Thought in a Large Pretrained Time Series Forecasting Model

Xue Wang, Tian Zhou, Jinyang Gao, Bolin Ding, Jingren Zhou

机构 * Alibaba-inc（阿里巴巴集团） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.AI、cs.LG

TeleMath: A Benchmark for Large Language Models in Telecom Mathematical Problem Solving

Vincenzo Colle, Mohamed Sana, Nicola Piovesan, Antonio De Domenico, Fadhel Ayed, Merouane Debbah

机构 * Paris Research Center, Huawei Technologies（巴黎研究中心，华为技术） 纠错

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL、cs.AI

Comments 6 pages

IntentionESC: An Intention-Centered Framework for Enhancing Emotional Support in Dialogue Systems

Xinjie Zhang, Wenxuan Wang, Qin Jin

机构 * School of Information, Renmin University of China（中国人民大学信息学院） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Comments ACL2025 findings

EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents

Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, Tong Zhang

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI

Comments Accepted to ICML 2025

Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models

Haoyang Li, Xuejia Chen, Zhanchao XU, Darian Li, Nicole Hu, Fei Teng, Yiming Li, Luyu Qiu, Chen Jason Zhang, Qing Li, Lei Chen

机构 * Hong Kong Polytechnic University（香港理工大学） ； Hong Kong University of Science and Technology（香港科技大学） ； The Chinese University of Hong Kong（香港中文大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted by ACL 2025

Spurious Correlations and Beyond: Understanding and Mitigating Shortcut Learning in SDOH Extraction with Large Language Models

Fardin Ahsan Sakib, Ziwei Zhu, Karen Trister Grace, Meliha Yetisgen, Ozlem Uzuner

机构 * Department of Computer Science（计算机科学系） ； School of Nursing（护理学院） ； Department of Information Sciences and Technology（信息科学与技术系） ； Department of Biomedical Informatics & Medical Education（生物医学信息学与医学教育系） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias

Ruiquan Huang, Yingbin Liang, Jing Yang

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.LG

Comments accepted by ICML 2025

TheoremExplainAgent: Towards Video-based Multimodal Explanations for LLM Theorem Understanding

Max Ku, Thomas Chong, Jonathan Leung, Krish Shah, Alvin Yu, Wenhu Chen

机构 * University of Waterloo（滑铁卢大学） ； Votee AI ； Vector Institute（向量研究所） 纠错

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI

Comments accepted to ACL 2025 main, camera ready

Language Models Benefit from Preparation with Elicited Knowledge

Jiacan Yu, Hannah An, Lenhart K. Schubert

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

LLMs Outperform Experts on Challenging Biology Benchmarks

Lennart Justen

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.AI、cs.LG

Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators

Yilun Zhou, Austin Xu, Peifeng Wang, Caiming Xiong, Shafiq Joty

专题命中 推理评测 ：reasoning(abstract)；math reasoning(abstract)；分类 cs.CL、cs.LG

Comments ICML 2025. The first two authors contributed equally. The codebase is at https://github.com/SalesforceAIResearch/jetts-benchmark

Quantifying the Capability Boundary of DeepSeek Models: An Application-Driven Performance Analysis

Kaikai Zhao, Zhaoxiang Liu, Xuejiao Lei, Jiaojiao Zhao, Zhenhong Long, Zipeng Wang, Ning Wang, Meijuan An, Qingliang Meng, Peijun Yang, Minjie Hua, Chaoyang Ma, Wen Liu, Kai Wang, Shiguo Lian

机构 * Unicom Data Intelligence（中国unicom数据智能） ； Data Science & Artificial Intelligence Research Institute（数据科学与人工智能研究院） ； China Unicom（中国unicom） 纠错

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.AI、cs.LG

Benchmarking Generative AI for Scoring Medical Student Interviews in Objective Structured Clinical Examinations (OSCEs)

Jadon Geathers, Yann Hicke, Colleen Chan, Niroop Rajashekar, Justin Sewell, Susannah Cornes, Rene F. Kizilcec, Dennis Shung

机构 * Cornell University（康奈尔大学） ； Yale University（耶鲁大学） ； University of California San Francisco（旧金山大学） 纠错

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

Comments 12 pages + 3 pages of references, 4 figures

Keep the General, Inject the Specific: Structured Dialogue Fine-Tuning for Knowledge Injection without Catastrophic Forgetting

Yijie Hong, Xiaofei Yin, Xinzhong Wang, Yi Tu, Ya Guo, Sufeng Duan, Weiqiang Wang, Lingyong Fang, Depeng Wang, Huijia Zhu

机构 * Shanghai Jiao Tong University（上海交通大学） ； Ant Security Lab, Ant Group（蚂蚁集团安全实验室） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Comments 13 pages, 3 figures

ZeroSumEval: Scaling LLM Evaluation with Inter-Model Competition

Haidar Khan, Hisham A. Alyahya, Yazeed Alnumay, M Saiful Bari, Bülent Yener

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI

ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competition

Hisham A. Alyahya, Haidar Khan, Yazeed Alnumay, M Saiful Bari, Bülent Yener

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL、cs.AI

The Future of MLLM Prompting is Adaptive: A Comprehensive Experimental Evaluation of Prompt Engineering Methods for Robust Multimodal Performance

Anwesha Mohanty, Venkatesh Balavadhani Parthasarathy, Arsalan Shahid

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

HDLCoRe: A Training-Free Framework for Mitigating Hallucinations in LLM-Generated HDL

Heng Ping, Shixuan Li, Peiyu Zhang, Anzhe Cheng, Shukai Duan, Nikos Kanakaris, Xiongye Xiao, Wei Yang, Shahin Nazarian, Andrei Irimia, Paul Bogdan

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

VeriLA: A Human-Centered Evaluation Framework for Interpretable Verification of LLM Agent Failures

Yoo Yeon Sung, Hannah Kim, Dan Zhang

专题命中 推理评测 ：reasoning(abstract)；verifier(abstract)；分类 cs.CL、cs.AI

General Scales Unlock AI Evaluation with Explanatory and Predictive Power

Lexin Zhou, Lorenzo Pacchiardi, Fernando Martínez-Plumed, Katherine M. Collins, Yael Moros-Daval, Seraphina Zhang, Qinlin Zhao, Yitian Huang, Luning Sun, Jonathan E. Prunty, Zongqian Li, Pablo Sánchez-García, Kexin Jiang Chen, Pablo A. M. Casares, Jiyun Zu, John Burden, Behzad Mehrbakhsh, David Stillwell, Manuel Cebrian, Jindong Wang, Peter Henderson, Sherry Tongshuang Wu, Patrick C. Kyllonen, Lucy Cheke, Xing Xie, José Hernández-Orallo

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Text2Zinc: A Cross-Domain Dataset for Modeling Optimization and Satisfaction Problems in MiniZinc

Akash Singirikonda, Serdar Kadioglu, Karthik Uppuluri

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Bot Wars Evolved: Orchestrating Competing LLMs in a Counterstrike Against Phone Scams

Nardine Basta, Conor Atkins, Dali Kaafar

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Journal ref Pacific-Asia Conference on Knowledge Discovery and Data Mining, PAKDD 2025

Benchmarking Large Language Models on Multiple Tasks in Bioinformatics NLP with Prompting

Jiyue Jiang, Pengan Chen, Jiuming Wang, Dongchen He, Ziqin Wei, Liang Hong, Licheng Zong, Sheng Wang, Qinze Yu, Zixian Ma, Yanyu Chen, Yimin Fan, Xiangyu Shi, Jiawei Sun, Chuan Wu, Yu Li

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL、cs.AI

NYT-Connections: A Deceptively Simple Text Classification Task that Stumps System-1 Thinkers

Angel Yahir Loredo Lopez, Tyler McDonald, Ali Emami

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Comments 5 pages (excluding references), Published at Coling 2025, Best Dataset Paper Award