大模型推理能力

VLM@school -- Evaluation of AI image understanding on German middle school knowledge

René Peinl, Vincent Tischler

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Peinl, René; Tischler, Vincent (2025): VLM@school - Evaluation of AI image understanding on German middle school knowledge. Future Technologies Conference (FTC) 2025, Munich, Germany 2025 (accepted)

Large Language Models Acing Chartered Accountancy

Jatin Gupta, Akhil Sharma, Saransh Singhania, Mohammad Adnan, Sakshi Deo, Ali Imam Abidi, Keshav Gupta

机构 * Sharda University（莎达大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted for publication at MoStart 2025: International Conference on Digital Transformation in Education and Applications of Artificial Intelligence, Bosnia and Herzegovina, 2025

"I know myself better, but not really greatly": How Well Can LLMs Detect and Explain LLM-Generated Texts?

Jiazhou Ji, Jie Guo, Weidong Qiu, Zheng Huang, Yang Xu, Xinru Lu, Xiaoyu Jiang, Ruizhe Li, Shujun Li

机构 * School of Cyber Science and Engineering, Shanghai Jiao Tong University, China（上海交通大学信息科学与工程学院） ； Department of Computing Science, University of Aberdeen, UK（阿伯丁大学计算机科学系） ； Institute of Cyber Security for Society (iCSS) & School of Computing, University of Kent, UK（社会网络安全研究所（iCSS）及肯特大学计算机学院） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Under review

Evaluating Causal Explanation in Medical Reports with LLM-Based and Human-Aligned Metrics

Yousang Cho, Key-Sun Choi

机构 * Department of Medical Engineering, Konyang University（康山大学医学工程系） ； Department of Artificial Intelligence, Konyang University（康山大学人工智能系） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 9 pages, presented at LLM4Eval Workshop, SIGIR 2025 Padova, Italy, July 17, 2025

Mental Health Equity in LLMs: Leveraging Multi-Hop Question Answering to Detect Amplified and Silenced Perspectives

Batool Haider, Atmika Gorti, Aman Chadha, Manas Gaur

机构 * University of Maryland Baltimore County（马里兰大学巴尔的摩分校） ； Stanford University and Amazon AI（斯坦福大学和亚马逊人工智能） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 19 Pages, 7 Figures, 4 Tables (Note: Under Review)

Supernova Event Dataset: Interpreting Large Language Models' Personality through Critical Event Analysis

Pranav Agarwal, Ioana Ciucă

机构 * Stanford University（斯坦福大学） ； Mila, Quebec AI Institute（魁北克AI研究院） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted at Actionable Interpretability Workshop at ICML 2025

Evaluating LLMs with Multiple Problems at once

Zhengxiang Wang, Jordan Kodner, Owen Rambow

机构 * Department of Linguistics & Institute for Advanced Computational Science（语言学系及高级计算科学研究所） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 22 pages, 9 figures, 12 tables

StoryWriter: A Multi-Agent Framework for Long Story Generation

Haotian Xia, Hao Peng, Yunjia Qi, Xiaozhi Wang, Bin Xu, Lei Hou, Juanzi Li

专题命中 推理评测 ：planning(abstract)；分类 cs.CL、cs.AI

Finance Language Model Evaluation (FLaME)

Glenn Matlin, Mika Okamoto, Huzaifa Pardawala, Yang Yang, Sudheer Chava

机构 * Georgia Institute of Technology（佐治亚理工学院） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Compiler-R1: Towards Agentic Compiler Auto-tuning with Reinforcement Learning

Haolin Pan, Hongyu Lin, Haoran Luo, Yang Liu, Kaichun Yao, Libo Zhang, Mingjie Xing, Yanjun Wu

机构 * University of Chinese Academy of Sciences（中国科学院大学） ； Institute of Software Chinese Academy of Sciences（中国科学院软件研究所） ； Beijing University of Posts and Telecommunications（北京邮电大学） ； Hangzhou Institute for Advanced Study at UCAS（UCAS杭州高等研究院） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Batayan: A Filipino NLP benchmark for evaluating Large Language Models

Jann Railey Montalan, Jimson Paulo Layacan, David Demitri Africa, Richell Isaiah Flores, Michael T. Lopez, Theresa Denise Magsajo, Anjanette Cayabyab, William Chandra Tjhi

机构 * AI Singapore（AI新加坡） ； National University of Singapore（新加坡国立大学） ； Ateneo de Manila University（马尼拉大学） ； University of Cambridge（剑桥大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted to ACL 2025 (Main Conference)

Core Knowledge Deficits in Multi-Modal Language Models

Yijiang Li, Qingying Gao, Tianwei Zhao, Bingyang Wang, Haoran Sun, Haiyun Lyu, Robert D. Hawkins, Nuno Vasconcelos, Tal Golan, Dezhi Luo, Hokin Deng

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted by ICML 2025. Project page at https://williamium3000.github.io/core-knowledge and code is available at https://github.com/williamium3000/core-knowledge

J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization

Austin Xu, Yilun Zhou, Xuan-Phi Nguyen, Caiming Xiong, Shafiq Joty

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 25 pages, 4 figures, 6 tables. Updated with code and benchmark

InsertRank: LLMs can reason over BM25 scores to Improve Listwise Reranking

Rahul Seetharaman, Kaustubh D. Dhole, Aman Bansal

机构 * Emory University（埃默里大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints

Zekun Li, Shinda Huang, Jiangtian Wang, Nathan Zhang, Antonis Antoniades, Wenyue Hua, Kaijie Zhu, Sirui Zeng, Chi Wang, William Yang Wang, Xifeng Yan

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Code, data, and over 24k agent trajectories are released at https://github.com/Leezekun/SOPBench

PredictaBoard: Benchmarking LLM Score Predictability

Lorenzo Pacchiardi, Konstantinos Voudouris, Ben Slater, Fernando Martínez-Plumed, José Hernández-Orallo, Lexin Zhou, Wout Schellaert

机构 * Leverhulme Centre for the Future of Intelligence, University of Cambridge（未来智能中心，剑桥大学） ； Institute for Human-Centered AI, Helmholtz Zentrum Munich（人本人工智能研究所，慕尼黑赫尔姆霍尔茨中心） ； VRAIN, Universitat Politècnica de València（VRAIN，瓦伦西亚理工大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted at ACL Findings 2025

TimeMaster: Training Time-Series Multimodal LLMs to Reason via Reinforcement Learning

Junru Zhang, Lang Feng, Xu Guo, Yuhan Wu, Yabo Dong, Duanqing Xu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Comments Preprint

StoryBench: A Dynamic Benchmark for Evaluating Long-Term Memory with Multi Turns

Luanbo Wan, Weizhi Ma

机构 * Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China（人工智能产业研究院（AIR），清华大学，北京，中国） ； University of Electronic Science and Technology of China, Chengdu, China（电子科技大学，成都，中国） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 13pages, 8 figures, 4 tables

WereWolf-Plus: An Update of Werewolf Game setting Based on DSGBench

Xinyuan Xia, Yuanyi Song, Haomin Ma, Jinyu Cai

机构 * Shanghai Jiao Tong University（上海交通大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

RealFactBench: A Benchmark for Evaluating Large Language Models in Real-World Fact-Checking

Shuo Yang, Yuqin Dai, Guoqing Wang, Xinran Zheng, Jinfeng Xu, Jinze Li, Zhenzhe Ying, Weiqiang Wang, Edith C. H. Ngai

机构 * The University of Hong Kong（香港大学） ； Tsinghua University（清华大学） ； University College London（伦敦大学学院） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

AgentCourt: Simulating Court with Adversarial Evolvable Lawyer Agents

Guhong Chen, Liyang Fan, Zihan Gong, Nan Xie, Zixuan Li, Ziqiang Liu, Chengming Li, Qiang Qu, Hamid Alinejad-Rokny, Shiwen Ni, Min Yang

机构 * Southern University of Science and Technology（南方科技大学） ； Shenzhen Key Laboratory for High Performance Data Mining, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳高性能数据挖掘重点实验室，深圳先进技术研究所，中国科学院） ； SIAT-DELI AI and Law Joint Lab（SIAT-DELI人工智能与法律联合实验室） ； Shenzhen MSU-BIT University（深圳MSU-BIT大学） ； UNSW Sydney（新南威尔士大学悉尼分校） ； SUAT（南方科技大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

TrustGLM: Evaluating the Robustness of GraphLLMs Against Prompt, Text, and Structure Attacks

Qihai Zhang, Xinyue Sheng, Yuanfu Sun, Qiaoyu Tan

机构 * New York University（纽约大学） ； New York University Shanghai（纽约大学上海） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Comments 12 pages, 5 figures, in KDD 2025

Trustworthy AI for Medicine: Continuous Hallucination Detection and Elimination with CHECK

Carlos Garcia-Fernandez, Luis Felipe, Monique Shotande, Muntasir Zitu, Aakash Tripathi, Ghulam Rasool, Issam El Naqa, Vivek Rudrapatna, Gilmer Valdes

机构 * Department of Machine Learning, Moffitt Cancer Center（机器学习部门，莫菲特癌症中心） ； Department of Medicine, University of California San Francisco（医学部门，加州大学旧金山分校） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

KokushiMD-10: Benchmark for Evaluating Large Language Models on Ten Japanese National Healthcare Licensing Examinations

Junyu Liu, Kaiqi Yan, Tianyang Wang, Qian Niu, Momoko Nagai-Tanima, Tomoki Aoyama

机构 * Kyoto University（京都大学） ； The Hong Kong University of Science and Technology（香港科技大学） ； Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学） ； The University of Tokyo（东京大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 9pages, 3 figures

CodeMirage: A Multi-Lingual Benchmark for Detecting AI-Generated and Paraphrased Source Code from Production-Level LLMs

Hanxi Guo, Siyuan Cheng, Kaiyuan Zhang, Guangyu Shen, Xiangyu Zhang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG

MEDDxAgent: A Unified Modular Agent Framework for Explainable Automatic Differential Diagnosis

Daniel Rose, Chia-Chien Hung, Marco Lepri, Israa Alqassem, Kiril Gashteovski, Carolin Lawrence

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments ACL 2025 (main)

Slimming Down LLMs Without Losing Their Minds

Qingda, Mai

机构 * University of Waterloo（滑铁卢大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 10 pages

EgoNormia: Benchmarking Physical Social Norm Understanding

MohammadHossein Rezaei, Yicheng Fu, Phil Cuvin, Caleb Ziems, Yanzhe Zhang, Hao Zhu, Diyi Yang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments V4, fixes to title and formatting

Prompt-based Depth Pruning of Large Language Models

Juyun Wee, Minjae Park, Jaeho Lee

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Project: https://jwee01.github.io/PuDDing/ Code: https://github.com/tada0347/PuDDing

VerIF: Verification Engineering for Reinforcement Learning in Instruction Following

Hao Peng, Yunjia Qi, Xiaozhi Wang, Bin Xu, Lei Hou, Juanzi Li

机构 * Department of Computer Science and Technology, Tsinghua University（计算机科学与技术系，清华大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 16 pages, 8 figures