大模型推理能力

Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and Benchmarking Multimodal LLM Agents

Yaxin Luo, Zhaoyi Li, Jiacheng Liu, Jiacheng Cui, Xiaohan Zhao, Zhiqiang Shen

机构 * VILA Lab, MBZUAI（VILA实验室，MBZUAI） ； MetaAgentX 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Code at: https://github.com/MetaAgentX/OpenCaptchaWorld

Revisiting Uncertainty Estimation and Calibration of Large Language Models

Linwei Tao, Yi-Fan Yeh, Minjing Dong, Tao Huang, Philip Torr, Chang Xu

机构 * School of Computer Science University of Sydney（悉尼大学计算机科学学院） ； City University of Hong Kong（香港城市大学） ； Shanghai Jiao Tong University（上海交通大学） ； Department of Engineering Science University of Oxford（牛津大学工程科学系） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

VAU-R1: Advancing Video Anomaly Understanding via Reinforcement Fine-Tuning

Liyun Zhu, Qixiang Chen, Xi Shen, Xiaodong Cun

机构 * Australian National University（澳大利亚国立大学） ； GVC Lab, Great Bay University（大湾大学GVC实验室） ； Intellindust AI Lab（Intellindust AI实验室） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)

VulBinLLM: LLM-powered Vulnerability Detection for Stripped Binaries

Nasir Hussain, Haohan Chen, Chanh Tran, Philip Huang, Zhuohao Li, Pravir Chugh, William Chen, Ashish Kundu, Yuan Tian

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)

What Makes a Scene ? Scene Graph-based Evaluation and Feedback for Controllable Generation

Zuyao Chen, Jinlin Wu, Zhen Lei, Chang Wen Chen

机构 * The Hong Kong Polytechnic University（香港理工大学） ； Institute of Automation, CAS（中国科学院自动化研究所） 纠错

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)

MT$^{3}$: Scaling MLLM-based Text Image Machine Translation via Multi-Task Reinforcement Learning

Zhaopeng Feng, Yupu Liang, Shaosheng Cao, Jiayuan Su, Jiahan Ren, Zhe Xu, Yao Hu, Wenxuan Huang, Jian Wu, Zuozhu Liu

机构 * Zhejiang University（浙江大学） ； University of Chinese Academy of Sciences（中国科学院大学） ； Xiaohongshu Inc.（小红书公司） ； East China Normal University（华东师范大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Work in progress

ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models

Benjamin Clavié, Florian Brand

机构 * Artificial Intelligence and Intelligent Information Systems, University of Trier（人工智能与智能信息系统，特里尔大学） ； German Research Center for Artificial Intelligence (DFKI), Trier（德国人工智能研究中心（DFKI），特里尔） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

SMART: Self-Aware Agent for Tool Overuse Mitigation

Cheng Qian, Emre Can Acikgoz, Hongru Wang, Xiusi Chen, Avirup Sil, Dilek Hakkani-Tür, Gokhan Tur, Heng Ji

机构 * University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） ； IBM Research AI（IBM人工智能研究院） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 18 pages, 11 tables, 7 figures, ACL 2025 Findings

Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning

Guanting Dong, Yifei Chen, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Yutao Zhu, Hangyu Mao, Guorui Zhou, Zhicheng Dou, Ji-Rong Wen

机构 * Renmin University of China（中国人民大学） ； BAAI（北京人工智能研究院） ； Kuaishou Technology（快手科技） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Working in progress

ReCopilot: Reverse Engineering Copilot in Binary Analysis

Guoqiang Chen, Huiqi Sun, Daguang Liu, Zhiqi Wang, Qiang Wang, Bin Yin, Lu Liu, Lingyun Ying

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)

Leveraging LLM Inconsistency to Boost Pass@k Performance

Uri Dalal, Meirav Segal, Zvika Ben-Haim, Dan Lahav, Omer Nevo

机构 * Pattern Labs 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations

Katie Matton, Robert Osazuwa Ness, John Guttag, Emre Kıcıman

机构 * MIT（麻省理工学院） ； Microsoft Research（微软研究院） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 66 pages, 14 figures, 40 tables; ICLR 2025 (spotlight) camera ready

M-RewardBench: Evaluating Reward Models in Multilingual Settings

Srishti Gureja, Lester James V. Miranda, Shayekh Bin Islam, Rishabh Maheshwary, Drishti Sharma, Gusti Winata, Nathan Lambert, Sebastian Ruder, Sara Hooker, Marzieh Fadaee

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 16 pages, 6 figures, 10 tables. Website: https://m-rewardbench.github.io/ , Updated results with latest models. Added more author information

Bridging Generative and Discriminative Learning: Few-Shot Relation Extraction via Two-Stage Knowledge-Guided Pre-training

Quanjiang Guo, Jinchuan Zhang, Sijie Wang, Ling Tian, Zhao Kang, Bin Yan, Weidong Xiao

机构 * University of Electronic Science and Technology of China（电子科技大学） ； Nanyang Technological University（南洋理工大学） ； Information Engineering University（信息工程大学） ； National University of Defense Technology（国防科技大学） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 13 pages, 6 figures, Appear on IJCAI 2025

Enhancing LLM Evaluations: The Garbling Trick

William F. Bradley

机构 * Mirabolic Consulting（Mirabolic咨询公司） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 15 pages, 4 figures

Spec2Assertion: Automatic Pre-RTL Assertion Generation using Large Language Models with Progressive Regularization

Fenghua Wu, Evan Pan, Rahul Kande, Michael Quinn, Aakash Tyagi, David Kebo Houngninou, Jeyavijayan Rajendran, Jiang Hu

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)

Comments 8 pages, 7 figures

Chronocept: Instilling a Sense of Time in Machines

Krish Goel, Sanskar Pandey, KS Mahadevan, Harsh Kumar, Vishesh Khadaria

机构 * ProjectEndgame 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 20 pages, 8 figures, 18 tables

From First Draft to Final Insight: A Multi-Agent Approach for Feedback Generation

Jie Cao, Chloe Qianhui Zhao, Xian Chen, Shuman Wang, Christian Schunn, Kenneth R. Koedinger, Jionghao Lin

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)

Comments 14 pages, to be published at the 26th International Conference on Artificial Intelligence in Education (AIED '25)

Communicating Activations Between Language Model Agents

Vignav Ramesh, Kenneth Li

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICML 2025

RESAnything: Attribute Prompting for Arbitrary Referring Segmentation

Ruiqi Wang, Hao Zhang

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)

Comments 42 pages, 31 figures. For more details: https://suikei-wang.github.io/RESAnything/

R&B: Domain Regrouping and Data Mixture Balancing for Efficient Foundation Model Training

Albert Ge, Tzu-Heng Huang, John Cooper, Avi Trost, Ziyi Chu, Satya Sai Srinath Namburi GNVV, Ziyang Cai, Kendall Park, Nicholas Roberts, Frederic Sala

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Fact-Consistency Evaluation of Text-to-SQL Generation for Business Intelligence Using Exaone 3.5

Jeho Choi

机构 * LG Electronics（LG电子） 纠错

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 6 pages, 1 table

LELANTE: LEveraging LLM for Automated ANdroid TEsting

Shamit Fatin, Mehbubul Hasan Al-Quvi, Haz Sameen Shahgir, Sukarna Barua, Anindya Iqbal, Sadia Sharmin, Md. Mostofa Akbar, Kallol Kumar Pal, A. Asif Al Rashid

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)

Comments 6 pages, 4 figures, 29th International Conference on Evaluation and Assessment in Software Engineering (EASE)

Benchmarking large language models for biomedical natural language processing applications and recommendations

Qingyu Chen, Yan Hu, Xueqing Peng, Qianqian Xie, Qiao Jin, Aidan Gilson, Maxwell B. Singer, Xuguang Ai, Po-Ting Lai, Zhizheng Wang, Vipina Kuttichi Keloth, Kalpana Raja, Jiming Huang, Huan He, Fongci Lin, Jingcheng Du, Rui Zhang, W. Jim Zheng, Ron A. Adelman, Zhiyong Lu, Hua Xu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Journal ref Nature Communications; 2025

FinNLI: Novel Dataset for Multi-Genre Financial Natural Language Inference Benchmarking

Jabez Magomere, Elena Kochkina, Samuel Mensah, Simerjot Kaur, Charese H. Smiley

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Causality for Natural Language Processing

Zhijing Jin

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments PhD Thesis 2024

Code2API: A Tool for Generating Reusable APIs from Stack Overflow Code Snippets

Yubo Mai, Zhipeng Gao, Xing Hu, Lingfeng Bao, Jingyuan Chen, Jianling Sun

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)

LiveBench: A Challenging, Contamination-Limited LLM Benchmark

Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, Micah Goldblum

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICLR 2025 Spotlight

SHIELD : An Evaluation Benchmark for Face Spoofing and Forgery Detection with Multimodal Large Language Models

Yichen Shi, Yuhao Gao, Yingxin Lai, Hongyang Wang, Jun Feng, Lei He, Jun Wan, Changsheng Chen, Zitong Yu, Xiaochun Cao

机构 * School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院） ； Department of Artificial Intelligence, Xiamen University（厦门大学人工智能系） ； School of Information Science and Technology, Shijiazhuang Tiedao University（石家庄铁道大学信息科学与技术学院） ； Electrical Engineering Department, UCLA（加州大学洛杉矶分校电子工程系） ； New Laboratory of Pattern Recognition(NLPR), Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别新实验室） ； College of Electronics and Information Engineering, Shenzhen University（深圳大学电子与信息工程学院） ； School of Computing and Information Technology, Great Bay University（广东东莞Great Bay大学计算与信息科技学院） ； School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络科学与技术学院） 纠错

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)

Comments Accepted by Visual Intelligence

KFinEval-Pilot: A Comprehensive Benchmark Suite for Korean Financial Language Understanding

Bokwang Hwang, Seonkyu Lim, Taewoong Kim, Yongjae Geun, Sunghyun Bang, Sohyun Park, Jihyun Park, Myeonggyu Lee, Jinwoo Lee, Yerin Kim, Jinsun Yoo, Jingyeong Hong, Jina Park, Yongchan Kim, Suhyun Kim, Younggyun Hahm, Yiseul Lee, Yejee Kang, Chanhyuk Yoon, Chansu Lee, Heeyewon Jeong, Jiyeon Lee, Seonhye Gu, Hyebin Kang, Yousang Cho, Hangyeol Yoo, KyungTae Lim

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI、cs.LG