AI Agent

Towards Outcome-Oriented, Task-Agnostic Evaluation of AI Agents

Waseem AlShikh, Muayad Sayed Ali, Brian Kennedy, Dmytro Mozolevskyi

机构 * Writer, Inc.（Writer公司） 纠错

专题命中 Agent评测 ：AI agent(title,abstract)；agent(abstract)；分类 cs.AI、cs.CL

SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models

Jingxuan Xu, Ken Deng, Weihao Li, Songwei Yu, Huaixi Tang, Haoyang Huang, Zhiyi Lai, Zizheng Zhan, Yanan Wu, Chenchen Zhang, Kepeng Lei, Yifan Yao, Xinping Lei, Wenqiang Zhu, Zongxian Feng, Han Li, Junqi Xiong, Dailin Li, Zuchen Gao, Kun Wu, Wen Xiang, Ziqi Zhan, Yuanxing Zhang, Wuxuan Gong, Ziyuan Gao, Guanxiang Wang, Yirong Xue, Mengtong Li, Mengfei Xie, Xiaojiang Zhang, Jinghui Wang, Wenhao Zhuang, Zheng Lin, Huiming Wang, Zhaoxiang Zhang, Yuqun Zhang, Haotian Zhang, Bin Chen, Jiaheng Liu

机构 * Kuaishou Technology（快手科技） ； Nanjing University（南京大学） 纠错

专题命中 Agent评测 ：agentic(title,abstract)；agent(abstract)；分类 cs.AI、cs.SE

Towards a Standard, Enterprise-Relevant Agentic AI Benchmark: Lessons from 5.5 billion tokens' worth of agentic AI evaluations

JV Roig

机构 * Kamiwaza AI 纠错

专题命中 Agent评测 ：agentic(title,abstract)；tool use(abstract)；分类 cs.AI

Comments 34 pages

Planetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languages

Max Zuo, Francisco Piedrahita Velez, Xiaochen Li, Michael L. Littman, Stephen H. Bach

机构 * Brown University（布朗大学） 纠错

专题命中 Agent评测 ：planning(title,abstract)；分类 cs.AI、cs.CL、cs.LG

Comments NAACL Main Conference 2025

Journal ref Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)

How Brittle is Agent Safety? Rethinking Agent Risk under Intent Concealment and Task Complexity

Zihan Ma, Dongsheng Zhu, Shudong Liu, Taolin Zhang, Junnan Liu, Qingqiu Li, Minnan Luo, Songyang Zhang, Kai Chen

机构 * School of Computer Science and Technology Xi’an Jiaotong University China（西安交通大学计算机科学与技术学院） ； Shanghai AI Laboratory（上海人工智能实验室） ； MOE KLINNS Lab Xi’an Jiaotong University China（西安交通大学教育部KLINNS实验室） 纠错

专题命中 Agent评测 ：agent(title,abstract)；分类 cs.CL

Endpoint Security Agent: A Comprehensive Approach to Real-time System Monitoring and Threat Detection

Srihari R, Ayesha Taranum, Karthik, Mohammed Usman Hussain

专题命中 Agent评测 ：agent(title,abstract)

Comments 6 pages, 4 figures, 4 tables

AIA Forecaster: Technical Report

Rohan Alur, Bradly C. Stadie, Daniel Kang, Ryan Chen, Matt McManus, Michael Rickert, Tyler Lee, Michael Federici, Richard Zhu, Dennis Fogerty, Hayley Williamson, Nina Lozinski, Aaron Linsky, Jasjeet S. Sekhon

机构 * Bridgewater AIA Labs（布里奇沃特AIA实验室） 纠错

专题命中 Agent评测 ：agent(abstract)；agentic(abstract)；分类 cs.AI

JaxRobotarium: Training and Deploying Multi-Robot Policies in 10 Minutes

Shalin Anand Jain, Jiazhen Liu, Siva Kailas, Harish Ravichandar

机构 * Georgia Institute of Technology（佐治亚理工学院） 纠错

专题命中 Agent评测 ：agent(abstract)；multi-agent(abstract)；分类 cs.LG

Comments 22 pages, 14 figures, 10 tables. https://github.com/GT-STAR-Lab/JaxRobotarium. Manuscript accepted for publication at the 9th Conference on Robot Learning (CoRL 2025), Seoul, Korea

ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents

Manasi Sharma, Chen Bo Calvin Zhang, Chaithanya Bandi, Clinton Wang, Ankit Aich, Huy Nghiem, Tahseen Rabbani, Ye Htet, Brian Jang, Sumana Basu, Aishwarya Balwani, Denis Peskoff, Marcos Ayestaran, Sean M. Hendryx, Brad Kenstler, Bing Liu

机构 * Scale AI ； University of Maryland（马里兰大学） ； University of Chicago（芝加哥大学） ； Washington University, St. Louis（圣路易斯华盛顿大学） ； McGill University（麦吉尔大学） ； University of California, Berkeley（加州大学伯克利分校） 纠错

专题命中 Agent评测 ：agent(abstract)；分类 cs.AI、cs.CL、cs.LG

Comments 27 pages, 21 figures, pre-print

From LLMs to Agents: A Comparative Evaluation of LLMs and LLM-based Agents in Security Patch Detection

Junxiao Han, Zheng Yu, Lingfeng Bao, Jiakun Liu, Yao Wan, Jianwei Yin, Shuiguang Deng, Song Han

专题命中 Agent评测 ：agent(abstract)；分类 cs.SE

Distributionally Robust Online Markov Game with Linear Function Approximation

Zewu Zheng, Yuanyuan Lin

机构 * Zewu Zheng, Yuanyuan Lin（作者） 纠错

专题命中 Agent评测 ：agent(abstract)；分类 cs.LG

Comments To be published in the Proceedings of AAAI 2026

Design, Results and Industry Implications of the World's First Insurance Large Language Model Evaluation Benchmark

Hua Zhou, Bing Ma, Yufei Zhang, Yi Zhao

专题命中 Agent评测 ：agent(abstract)；分类 cs.CL

Comments 16 pages, 11 models,1 set of evaluation framework,5 core dimensions, 54 sub-indicators, 14,430 high-quality questions

Intelligent Optimization of Multi-Parameter Micromixers Using a Scientific Machine Learning Framework

Meraj Hassanzadeh, Ehsan Ghaderi, Mohamad Ali Bijarchi, Siamak Kazemzadeh Hannani

专题命中 Agent评测 ：agent(abstract)；分类 cs.LG

How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective

Songsong Yu, Yuxin Chen, Hao Ju, Lianjie Jia, Fuxi Zhang, Shaofei Huang, Yuhan Wu, Rundi Cui, Binghao Ran, Zaibin Zhang, Zhedong Zheng, Zhipeng Zhang, Yifan Wang, Lin Song, Lijun Wang, Yanwei Li, Ying Shan, Huchuan Lu

机构 * ARC Lab, Tencent PCG（腾讯PCG部门ARC实验室） ； Shanghai Jiao Tong University（上海交通大学） ； University of Macau（澳门大学） ； Dalian University of Technology（大连理工大学） ； The Chinese University of Hong Kong（香港中文大学） 纠错

专题命中 Agent评测 ：planning(abstract)；分类 cs.AI

Comments a comprehensive visual spatial reasoning evaluation tool, 25 pages, 16 figures

MPMA: Preference Manipulation Attack Against Model Context Protocol

Zihan Wang, Rui Zhang, Yu Liu, Wenshu Fan, Wenbo Jiang, Qingchuan Zhao, Hongwei Li, Guowen Xu

专题命中 Agent评测 ：agent(abstract)；分类 cs.CL

Comments This is an extended version of the copyrighted publication at AAAI

Integrating Artificial Intelligence into Operating Systems: A Survey on Techniques, Applications, and Future Directions

Yifan Zhang, Xinkui Zhao, Ziying Li, Guanjie Cheng, Jianwei Yin, Lufei Zhang, Zuoning Chen

机构 * School of Software Technology, Zhejiang University（浙江大学软件学院） ； College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院） ； State Key Laboratory of Mathematical Engineering and Advanced Computing（数学工程与先进计算国家重点实验室） ； Chinese Academy of Engineering（中国工程院） ； College of Cyber Security, Jinan University（暨南大学网络安全学院） 纠错

专题命中 Agent评测 ：agent(abstract)；分类 cs.AI

Comments 68 pages,9 figures

Shocks Under Control: Taming Transonic Compressible Flow over an RAE2822 Airfoil with Deep Reinforcement Learning

Trishit Mondal, Ricardo Vinuesa, Ameya D. Jagtap

机构 * Aerospace Engineering Department, Worcester Polytechnic Institute（航空航天工程系，沃斯特理工学院） ； Department of Aerospace Engineering, University of Michigan（航空航天工程系，密歇根大学） 纠错

专题命中 Agent评测 ：agent(abstract)；分类 cs.LG

Comments 23 pages, 18 figures

Provably Efficient Sample Complexity for Robust CMDP

Sourav Ganguly, Arnob Ghosh

机构 * New Jersey Institute of Technology（新泽西理工学院） 纠错

专题命中 Agent评测 ：agent(abstract)；分类 cs.LG

RL-Exec: Impact-Aware Reinforcement Learning for Opportunistic Optimal Liquidation, Outperforms TWAP and a Book-Liquidity VWAP on BTC-USD Replays

Enzo Duflot, Stanislas Robineau

机构 * Enzo Duflot, Stanislas Robineau 纠错

专题命中 Agent评测 ：agent(abstract)；分类 cs.LG

Comments 8 pages main text, 3 appendix pages, 10 figures

Expert Incentives under Partially Contractible States

Zizhe Xia

专题命中 Agent评测 ：agent(abstract)