大模型推理能力

Putnam-AXIOM: A Functional and Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs

Aryan Gulati, Brando Miranda, Eric Chen, Emily Xia, Kai Fronsdal, Bruno Dumont, Elyas Obbad, Sanmi Koyejo

机构 * Department of Computer Science, Stanford University, Stanford, USA（计算机科学系，斯坦福大学，斯坦福，美国） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 27 pages total (10-page main paper + 17-page appendix), 12 figures, 6 tables. Submitted to ICML 2025 (under review)

Journal ref ICML 2025

MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning

Run-Ze Fan, Zengzhi Wang, Pengfei Liu

机构 * Shanghai Jiao Tong University（上海交通大学） ； SII ； GAIR Lab（GAIR实验室） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 39 pages; Github: https://github.com/GAIR-NLP/MegaScience; HF: https://huggingface.co/MegaScience

LongReasonArena: A Long Reasoning Benchmark for Large Language Models

Jiayu Ding, Shuming Ma, Lei Cui, Nanning Zheng, Furu Wei

机构 * IAIR, Xi’an Jiaotong University（IAIR，西安交通大学） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI

Evaluating Language Model Reasoning about Confidential Information

Dylan Sam, Alexander Robey, Andy Zou, Matt Fredrikson, J. Zico Kolter

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.LG

Comments 20 pages

MedVQA-TREE: A Multimodal Reasoning and Retrieval Framework for Sarcopenia Prediction

Pardis Moradbeiki, Nasser Ghadiri, Sayed Jalal Zahabi, Uffe Kock Wiil, Kristoffer Kittelmann Brockhattingen, Ali Ebrahimi

机构 * Department of Electrical and Computer Engineering, Isfahan University of Technology（电气与计算机工程系，伊斯法罕技术大学） ； SDU Health Informatics and Technology, The Maersk Mc-Kinney Moller Institute, University of Southern Denmark（南部丹麦大学健康信息学与技术，马士基麦金尼莫勒研究所） ； Geriatric Research Unit, Department of Clinical Research, University of Southern Denmark（老年医学研究单元，临床研究系，南部丹麦大学） 纠错

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.AI

Demonstrating specification gaming in reasoning models

Alexander Bondarenko, Denis Volk, Dmitrii Volkov, Jeffrey Ladish

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.AI

Comments Updated with o3 results, fixed fonts

Rule Synergy Analysis using LLMs: State of the Art and Implications

Bahar Bateni, Benjamin Pratt, Jim Whitehead

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL

Comments Submitted for publication at the IEEE Transactions on Games 2024, Special Issue on Large Language Models and Games (10 pages excluding appendix, 3 figures)

Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMs

Yao Fu, Xianxuan Long, Runchao Li, Haotian Yu, Mu Sheng, Xiaotian Han, Yu Yin, Pan Li

机构 * Case Western Reserve University（凯斯西储大学） ； Hangzhou Dianzi University（杭州电子科技大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.AI

Comments Accepted to EMNLP2025 main conference (poster)

Symphony: A Decentralized Multi-Agent Framework for Scalable Collective Intelligence

Ji Wang, Kashing Chen, Xinyuan Song, Ke Zhang, Lynn Ai, Eric Yang, Bill Shi

机构 * Gradient ； The Chinese University of Hong Kong（香港中文大学） ； Department of Computer Science and Computer Engineering, Waseda University（计算机科学与工程系，早稻田大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Fine-Tuning Vision-Language Models for Neutrino Event Analysis in High-Energy Physics Experiments

Dikshant Sagar, Kaiwen Yu, Alejandro Yankelevich, Jianming Bian, Pierre Baldi

机构 * Department of Computer Science University of California, Irvine（计算机科学系加州大学伊文斯顿分校） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Tracking World States with Language Models: State-Based Evaluation Using Chess

Romain Harang, Jason Naradowsky, Yaswitha Gujju, Yusuke Miyao

机构 * University of Tokyo（东京大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI

Comments Spotlight presentation at ICML 2025 Workshop on Assessing World Models

Step-Audio 2 Technical Report

Boyong Wu, Chao Yan, Chen Hu, Cheng Yi, Chengli Feng, Fei Tian, Feiyu Shen, Gang Yu, Haoyang Zhang, Jingbei Li, Mingrui Chen, Peng Liu, Wang You, Xiangyu Tony Zhang, Xingyuan Li, Xuerui Yang, Yayue Deng, Yechang Huang, Yuxin Li, Yuxin Zhang, Zhao You, Brian Li, Changyi Wan, Hanpeng Hu, Jiangjie Zhen, Siyu Chen, Song Yuan, Xuelin Zhang, Yimin Jiang, Yu Zhou, Yuxiang Yang, Bingxin Li, Buyun Ma, Changhe Song, Dongqing Pang, Guoqiang Hu, Haiyang Sun, Kang An, Na Wang, Shuli Gao, Wei Ji, Wen Li, Wen Sun, Xuan Wen, Yong Ren, Yuankai Ma, Yufan Lu, Bin Wang, Bo Li, Changxin Miao, Che Liu, Chen Xu, Dapeng Shi, Dingyuan Hu, Donghang Wu, Enle Liu, Guanzhe Huang, Gulin Yan, Han Zhang, Hao Nie, Haonan Jia, Hongyu Zhou, Jianjian Sun, Jiaoren Wu, Jie Wu, Jie Yang, Jin Yang, Junzhe Lin, Kaixiang Li, Lei Yang, Liying Shi, Li Zhou, Longlong Gu, Ming Li, Mingliang Li, Mingxiao Li, Nan Wu, Qi Han, Qinyuan Tan, Shaoliang Pang, Shengjie Fan, Siqi Liu, Tiancheng Cao, Wanying Lu, Wenqing He, Wuxun Xie, Xu Zhao, Xueqi Li, Yanbo Yu, Yang Yang, Yi Liu, Yifan Lu, Yilei Wang, Yuanhao Ding, Yuanwei Liang, Yuanwei Lu, Yuchu Luo, Yuhe Yin, Yumeng Zhan, Yuxiang Zhang, Zidong Yang, Zixin Zhang, Binxing Jiao, Daxin Jiang, Heung-Yeung Shum, Jiansheng Chen, Jing Li, Xiangyu Zhang, Yibo Zhu

机构 * StepFun Audio Team（StepFun音频团队） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL

Comments v3: Added introduction and evaluation results of Step-Audio 2 mini

Efficient Response Generation Strategy Selection for Fine-Tuning Large Language Models Through Self-Aligned Perplexity

Xuan Ren, Qi Chen, Lingqiao Liu

机构 * University of Adelaide（阿德莱德大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL

NPHardEval4V: Dynamic Evaluation of Large Vision-Language Models with Effects of Vision

Xiang Li, Wenyue Hua, Kaijie Zhu, Lingyao Li, Haoyang Ling, Jinkui Chi, Qi Dou, Jindong Wang, Yongfeng Zhang, Xin Ma, Lizhou Fan

机构 * Shandong University, School of Control Science ； Rutgers University, Department of Computer Science 110 Frelinghuysen Road Piscataway New Brunswick NJ USA 08854 ； University of California, Santa Barbara 1210 Cheadle Hall Santa Barbara California USA 93106 ； University of Michigan, School of Information 500 S State St Ann Arbor Michigan USA 48109 ； The Chinese University of Hong Kong, Department of Computer Science ； The College of William \& Mary, School of Computing, Data Sciences \& Physics 200 Stadium Dr Williamsburg Virginia USA 23185 ； The Chinese University of Hong Kong, Department of Psychiatry 9 Chuen On Rd Tai Po New Territories Hong Kong SAR, China 999077 ； Rutgers University, Department of Computer Science ； University of California, Santa Barbara ； University of Michigan, School of Information ； The College of William \& Mary, School of Computing, Data Sciences \& Physics ； The Chinese University of Hong Kong, Department of Psychiatry 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL

Comments 25 pages, 9 figures, 2 tables

Leveraging LLMs for Automated Translation of Legacy Code: A Case Study on PL/SQL to Java Transformation

Lola Solovyeva, Eduardo Carneiro Oliveira, Shiyu Fan, Alper Tuncay, Shamil Gareev, Andrea Capiluppi

专题命中 推理评测 ：reasoning(abstract)

GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Data

Shengliang Deng, Mi Yan, Songlin Wei, Haixin Ma, Yuxin Yang, Jiayi Chen, Zhiqi Zhang, Taoyu Yang, Xuheng Zhang, Wenhao Zhang, Heming Cui, Zhizheng Zhang, He Wang

专题命中 推理评测 ：chain-of-thought(abstract)