大模型推理能力

Incorporating Contextual Paralinguistic Understanding in Large Speech-Language Models

Qiongqiong Wang, Hardik B. Sailor, Jeremy H. M. Wong, Tianchi Liu, Shuo Sun, Wenyu Zhang, Muhammad Huzaifah, Nancy Chen, Ai Ti Aw

机构 * Institute for Infocomm Research (I 2 R), Agency for Science, Technology 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted at (ASRU 2025) 2025 IEEE Automatic Speech Recognition and Understanding Workshop

Schema Lineage Extraction at Scale: Multilingual Pipelines, Composite Evaluation, and Language-Model Benchmarks

Jiaqi Yin, Yi-Wei Chen, Meng-Lung Lee, Xiya Liu

机构 * Microsoft Redmond, WA（微软红木城） ； Antra. Inc.（Antra公司） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

DatasetResearch: Benchmarking Agent Systems for Demand-Driven Dataset Discovery

Keyu Li, Mohan Jiang, Dayuan Fu, Yunze Wu, Xiangkun Hu, Dequan Wang, Pengfei Liu

机构 * Shanghai Jiao Tong University（上海交通大学） ； SII ； GAIR 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Pretraining on the Test Set Is No Longer All You Need: A Debate-Driven Approach to QA Benchmarks

Linbo Cao, Jinman Zhao

机构 * Faculty of Mathematics University of Waterloo（数学系大学水疗） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 22 pages, 7 figures. Accepted to COLM 2025. Code available at: github.com/l6cao/Debate-Driven-Evaluation

MAATS: A Multi-Agent Automated Translation System Based on MQM Evaluation

George Wang, Jiaqian Hu, Safinah Ali

专题命中 推理评测 ：self-correction(abstract)；分类 cs.CL、cs.LG

GrandJury: A Collaborative Machine Learning Model Evaluation Protocol for Dynamic Quality Rubrics

Arthur Cho

机构 * Memoirji LLC（Memiorji公司） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Comments 14 pages (incl. arXiv cover), 1 table, code & dataset links inside. Open-source implementation available on PyPI (grandjury package) and GitHub. Dataset available on Hugging Face under CC-BY-4.0 license

The SMeL Test: A simple benchmark for media literacy in language models

Gustaf Ahdritz, Anat Kleiman

机构 * Kempner Institute, Harvard University（凯普纳研究所，哈佛大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG

SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers

Yanzheng Xiang, Hanqi Yan, Shuyin Ouyang, Lin Gui, Yulan He

机构 * King’s College London（伦敦国王学院） ； The Alan Turing Institute（艾伦·图灵研究所） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

λ: A Benchmark for Data-Efficiency in Long-Horizon Indoor Mobile Manipulation Robotics

Ahmed Jaafar, Shreyas Sundara Raman, Sudarshan Harithas, Yichen Wei, Sofia Juliani, Anneke Wernerfelt, Benedict Quartey, Ifrah Idrees, Jason Xinyu Liu, Stefanie Tellex

机构 * Brown University（布朗大学） ； Rutgers University（罗格斯大学） ； University of Pennsylvania（宾夕法尼亚大学） 纠错

专题命中 推理评测 ：planning(abstract)；分类 cs.AI、cs.LG

Comments Accepted to IROS 2025. Sudarshan Harithas and Yichen Wei contributed equally. 8 pages. 7 figures

Do They Understand Them? An Updated Evaluation on Nonbinary Pronoun Handling in Large Language Models

Xushuo Tang, Yi Ding, Zhengyi Yang, Yin Chen, Yongrui Gu, Wenke Yang, Mingchen Ju, Xin Cao, Yongfei Liu, Wenjie Zhang

机构 * The University of New South Wales（新南威尔士大学） ； University of Technology Sydney（技术大学悉尼分校） ； Euler AI 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product

Paul Albert, Frederic Z. Zhang, Hemanth Saratchandran, Anton van den Hengel, Ehsan Abbasnejad

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG

Comments To appear in ICCV 2025

LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning

Federico Cocchi, Nicholas Moratelli, Davide Caffagni, Sara Sarto, Lorenzo Baraldi, Marcella Cornia, Rita Cucchiara

机构 * University of Modena and Reggio Emilia（摩德纳和雷吉奥艾米利亚大学） ； University of Pisa（比萨大学） ； IIT-CNR（意大利国家研究 council（IIT）） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments ICCV 2025 Workshop on What is Next in Multimodal Foundation Models

PATENTWRITER: A Benchmarking Study for Patent Drafting with LLMs

Homaira Huda Shomee, Suman Kalyan Maity, Sourav Medya

机构 * University of Illinois Chicago（伊利诺伊大学香槟分校） 纠错

专题命中 推理评测 ：chain-of-thought(abstract)；分类 cs.CL、cs.LG

OWLViz: An Open-World Benchmark for Visual Question Answering

Thuy Nguyen, Dang Nguyen, Hoang Nguyen, Thuan Luong, Long Hoang Dang, Viet Dac Lai

机构 * Posts and Telecommunications Institute of Technology, Viet Nam（越南电信技术研究所） ； Adobe Research, USA（Adobe研究） ； University of Maryland, USA（美国马里兰大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG

Comments 8 pages + appendix

Embeddings to Diagnosis: Latent Fragility under Agentic Perturbations in Clinical LLMs

Raj Krishnan Vijayaraj

机构 * Independent Researcher（独立研究者） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition

Andy Zou, Maxwell Lin, Eliot Jones, Micha Nowak, Mateusz Dziemian, Nick Winter, Alexander Grattan, Valent Nathanael, Ayla Croft, Xander Davies, Jai Patel, Robert Kirk, Nate Burnikell, Yarin Gal, Dan Hendrycks, J. Zico Kolter, Matt Fredrikson

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

MMCircuitEval: A Comprehensive Multimodal Circuit-Focused Benchmark for Evaluating LLMs

Chenchen Zhao, Zhengyuan Shi, Xiangyu Wen, Chengjie Liu, Yi Liu, Yunhao Zhou, Yuxiang Zhao, Hefei Feng, Yinan Zhu, Gwok-Waa Wan, Xin Cheng, Weiyu Chen, Yongqi Fu, Chujie Chen, Chenhao Xue, Guangyu Sun, Ying Wang, Yibo Lin, Jun Yang, Ning Xu, Xi Wang, Qiang Xu

机构 * Department of Computer Science and Engineering, The Chinese University of Hong Kong（中国香港中文大学计算机科学与工程系） ； School of Electronic Science and Engineering, Nanjing University（南京大学电子科学与工程学院） ； School of Integrated Circuits, Peking University（北京大学集成电路学院） ； School of Intergrated Circuits, Southeast University（东南大学集成电路学院） ； School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院） ； Department of Computer Science and Technology, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术系） ； National Center of Technology Innovation for EDA（EDA技术创新国家中心） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Comments 10 pages, 1 figure, 5 tables. To appear in ICCAD 2025

OS-MAP: How Far Can Computer-Using Agents Go in Breadth and Depth?

Xuetian Chen, Yinghao Chen, Xinfeng Yuan, Zhuo Peng, Lu Chen, Yuekeng Li, Zhoujia Zhang, Yingqian Huang, Leyan Huang, Jiaqing Liang, Tianbao Xie, Zhiyong Wu, Qiushi Sun, Biqing Qi, Bowen Zhou

机构 * Fudan University（复旦大学） ； Shanghai AI Lab（上海人工智能实验室） ； Tsinghua University（清华大学） ； The University of Hong Kong（香港大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Work in progress

2048: Reinforcement Learning in a Delayed Reward Environment

Prady Saligram, Tanvir Bhathal, Robby Manihani

专题命中 推理评测 ：planning(abstract)；分类 cs.AI、cs.LG

Comments We found an issue with our result aggregation scripts: some evaluation logs were incomplete and others duplicated, causing incorrect numbers in tables and figures. Because these graphs and tables underpin key comparisons, we are withdrawing the paper to regenerate verified results

A Survey of Event Causality Identification: Taxonomy, Challenges, Assessment, and Prospects

Qing Cheng, Zefan Zeng, Xingchen Hu, Yuehang Si, Zhong Liu

机构 * Laboratory for Big Data and Decision（大数据与决策实验室） ； National University of Defense Technology（国防科技大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

ReMeREC: Relation-aware and Multi-entity Referring Expression Comprehension

Yizhi Hu, Zezhao Tian, Xingqun Qi, Chen Su, Bingkun Yang, Junhui Yin, Muyi Sun, Man Zhang, Zhenan Sun

机构 * Beijing University of Posts and Telecommunications（北京邮电大学） ； Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 15 pages, 7 figures

A Method for the Architecture of a Medical Vertical Large Language Model Based on Deepseek R1

Mingda Zhang, Jianglong Qin

机构 * School of Software, Yunnan University（云南大学软件学院） ； Yunnan Key Laboratory of Software Engineering, Yunnan University（云南软件工程重点实验室） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 14 pages, 1 figures

Small Edits, Big Consequences: Telling Good from Bad Robustness in Large Language Models

Altynbek Ismailov, Salia Asanova

机构 * Berkeley（伯克利） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

MEETI: A Multimodal ECG Dataset from MIMIC-IV-ECG with Signals, Images, Features and Interpretations

Deyun Zhang, Xiang Lan, Shijia Geng, Qinghao Zhao, Sumei Fan, Mengling Feng, Shenda Hong

机构 * HeartVoice Medical Technology（HeartVoice医疗科技） ； Saw Swee Hock School of Public Health and Institute of Data Science（Saw Swee Hock公共卫生学院和数据科学研究所） ； National University of Singapore（新加坡国立大学） ； Department of Cardiology, Peking University People’s Hospital（北京大学人民医院心内科） ； College of Integrative Chinese and Western Medicine, Anhui University of Chinese Medicine（安徽中医药大学整合中西医学学院） ； National Institute of Health Data Science, Peking University（北京大学国家健康数据科学研究院） ； Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

ChronoSense: Exploring Temporal Understanding in Large Language Models with Time Intervals of Events

Duygu Sezen Islakoglu, Jan-Christoph Kalo

机构 * Utrecht University（乌特勒支大学） ； University of Amsterdam（阿姆斯特丹大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG

Comments Accepted to ACL 2025. Results on a larger test set. 13 pages, 2 figures

Hierarchical Prompting Taxonomy: A Universal Evaluation Framework for Large Language Models Aligned with Human Cognitive Principles

Devichand Budagam, Ashutosh Kumar, Mahsa Khoshnoodi, Sankalp KJ, Vinija Jain, Aman Chadha

机构 * Indian Institute of Technology Kharagpur（印度理工学院Khargapur分校） ； Rochester Institute of Technology（罗切斯特理工学院） ； Researcher, Fatima Fellowship（研究员，Fatima fellowship） ； AI Institute, University of South Carolina（人工智能研究所，南卡罗来纳大学） ； Amazon GenAI（亚马逊生成人工智能） ； Stanford University（斯坦福大学） ； James Silberrad Brown Center for AI, San Diego State University（詹姆斯·西伯拉德·布朗人工智能中心，圣地亚哥州立大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 18 pages, 9 figures, KDD workshop on Prompt Optimization 2025

Subliminal Learning: Language models transmit behavioral traits via hidden signals in data

Alex Cloud, Minh Le, James Chua, Jan Betley, Anna Sztyber-Betley, Jacob Hilton, Samuel Marks, Owain Evans

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Retrieval-Augmented Clinical Benchmarking for Contextual Model Testing in Kenyan Primary Care: A Methodology Paper

Fred Mutisya, Shikoh Gitau, Christine Syovata, Diana Oigara, Ibrahim Matende, Muna Aden, Munira Ali, Ryan Nyotu, Diana Marion, Job Nyangena, Nasubo Ongoma, Keith Mbae, Elizabeth Wamicha, Eric Mibuari, Jean Philbert Nsengemana, Talkmore Chidede

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 29 pages, 6 figs, 6 tables. Companion methods paper forthcoming

In-Depth and In-Breadth: Pre-training Multimodal Language Models Customized for Comprehensive Chart Understanding

Wan-Cyuan Fan, Yen-Chun Chen, Mengchen Liu, Alexander Jacobson, Lu Yuan, Leonid Sigal

机构 * UBC（不列颠哥伦比亚大学） ； Microsoft（微软） ； Vector Institute for AI（人工智能向量研究所） ； CIFAR AI Chair（卡尔·弗雷德里克人工智能主席） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments arXiv admin note: substantial text overlap with arXiv:2407.14506

DUALRec: A Hybrid Sequential and Language Model Framework for Context-Aware Movie Recommendation

Yitong Li, Raoul Grasman

机构 * University of Amsterdam（阿姆斯特丹大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Comments 10 pages, 5 figures