AI Agent

Diverse And Private Synthetic Datasets Generation for RAG evaluation: A multi-agent framework

Ilias Driouich, Hongliu Cao, Eoin Thomas

机构 * AMADEUS France（AMADEUS法国） 纠错

专题命中 Agent评测 ：agent(title,abstract)；multi-agent(title,abstract)；分类 cs.AI、cs.CL

Comments ECAI 2025 TRUST AI workshop

Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition

Andy Zou, Maxwell Lin, Eliot Jones, Micha Nowak, Mateusz Dziemian, Nick Winter, Alexander Grattan, Valent Nathanael, Ayla Croft, Xander Davies, Jai Patel, Robert Kirk, Nate Burnikell, Yarin Gal, Dan Hendrycks, J. Zico Kolter, Matt Fredrikson

专题命中 Agent评测 ：agent(title,abstract)；AI agent(title,abstract)；分类 cs.AI、cs.CL

Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents

Tianyi Men, Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao

机构 * The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences（认知与决策智能复杂系统重点实验室，自动化研究所，中国科学院） ； School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学） 纠错

专题命中 Agent评测 ：agent(title,abstract)；planning(title,abstract)；分类 cs.AI、cs.CL

Comments ACL 2025 Main

Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity

Hangfan Zhang, Zhiyao Cui, Jianhao Chen, Xinrun Wang, Qiaosheng Zhang, Zhen Wang, Dinghao Wu, Shuyue Hu

机构 * Pennsylvania State University（宾夕法尼亚州立大学） ； Northwestern Polytechnical University（西北工业大学） ； Singapore Management University（新加坡管理学院） ； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室） ； Nanjing University（南京大学） 纠错

专题命中 Agent评测 ：agent(title,abstract)；multi-agent(title,abstract)；分类 cs.CL、cs.LG

Comments This position paper takes a critical view of the status quo of MAD research, and outline multiple potential directions to improve MAD

POGEMA: A Benchmark Platform for Cooperative Multi-Agent Pathfinding

Alexey Skrynnik, Anton Andreychuk, Anatolii Borzilov, Alexander Chernyavskiy, Konstantin Yakovlev, Aleksandr Panov

专题命中 Agent评测 ：agent(title,abstract)；multi-agent(title,abstract)；分类 cs.AI、cs.LG

Comments Published as a conference paper at The International Conference on Learning Representations 2025

MAP: Evaluation and Multi-Agent Enhancement of Large Language Models for Inpatient Pathways

Zhen Chen, Zhihao Peng, Xusheng Liang, Cheng Wang, Peigan Liang, Linsheng Zeng, Minjie Ju, Yixuan Yuan

专题命中 Agent评测 ：agent(title,abstract)；multi-agent(title,abstract)；分类 cs.AI、cs.CL

M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluation

Zhaopeng Feng, Jiayuan Su, Jiamei Zheng, Jiahan Ren, Yan Zhang, Jian Wu, Hongwei Wang, Zuozhu Liu

专题命中 Agent评测 ：agent(title,abstract)；multi-agent(title,abstract)；分类 cs.AI、cs.CL

Comments Code and data are available at https://github.com/SU-JIAYUAN/M-MAD

Multi-Agent Simulator Drives Language Models for Legal Intensive Interaction

Shengbin Yue, Ting Huang, Zheng Jia, Siyuan Wang, Shujun Liu, Yun Song, Xuanjing Huang, Zhongyu Wei

专题命中 Agent评测 ：agent(title,abstract)；multi-agent(title,abstract)；分类 cs.AI、cs.CL

Comments Accepted by NAACL 2025

Personality Modeling for Persuasion of Misinformation using AI Agent

Qianmin Lou, Wentao Xu

专题命中 Agent评测 ：agent(title,abstract)；AI agent(title,abstract)；分类 cs.AI、cs.CL

TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraft

Qian Long, Zhi Li, Ran Gong, Ying Nian Wu, Demetri Terzopoulos, Xiaofeng Gao

专题命中 Agent评测 ：agent(title,abstract)；multi-agent(title,abstract)；分类 cs.AI、cs.CL

ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination

Xihuai Wang, Shao Zhang, Wenhao Zhang, Wentao Dong, Jingxiao Chen, Ying Wen, Weinan Zhang

专题命中 Agent评测 ：agent(title,abstract)；multi-agent(title,abstract)；分类 cs.AI、cs.LG

Comments Accepted in NeurIPS 2024 Dataset and Benchmark Track

FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learning

Wenzhe Li, Zihan Ding, Seth Karten, Chi Jin

专题命中 Agent评测 ：agent(title,abstract)；multi-agent(title,abstract)；分类 cs.AI、cs.LG

Comments ICML 2024

FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Models

Hongyang Yang, Boyu Zhang, Neng Wang, Cheng Guo, Xiaoli Zhang, Likun Lin, Junlin Wang, Tianyu Zhou, Mao Guan, Runjia Zhang, Christina Dan Wang

专题命中 Agent评测 ：agent(title,abstract)；AI agent(title,abstract)；分类 cs.CL、cs.LG

Comments FinRobot Whitepaper V1.0

MATEval: A Multi-Agent Discussion Framework for Advancing Open-Ended Text Evaluation

Yu Li, Shenyu Zhang, Rui Wu, Xiutian Huang, Yongrui Chen, Wenhao Xu, Guilin Qi, Dehai Min

专题命中 Agent评测 ：agent(title,abstract)；multi-agent(title,abstract)；分类 cs.AI、cs.CL

Comments This paper has been accepted as a long paper presentation by DASFAA 2024 Industrial Track

Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world

Eugene Vinitsky, Nathan Lichtlé, Xiaomeng Yang, Brandon Amos, Jakob Foerster

专题命中 Agent评测 ：agent(title,abstract)；multi-agent(title,abstract)；分类 cs.AI、cs.LG

CH-MARL: A Multimodal Benchmark for Cooperative, Heterogeneous Multi-Agent Reinforcement Learning

Vasu Sharma, Prasoon Goyal, Kaixiang Lin, Govind Thattai, Qiaozi Gao, Gaurav S. Sukhatme

专题命中 Agent评测 ：agent(title,abstract)；multi-agent(title,abstract)；分类 cs.AI、cs.LG

The AI Arena: A Framework for Distributed Multi-Agent Reinforcement Learning

Edward W. Staley, Corban G. Rivera, Ashley J. Llorens

专题命中 Agent评测 ：agent(title,abstract)；multi-agent(title,abstract)；分类 cs.AI、cs.LG

Arena: a toolkit for Multi-Agent Reinforcement Learning

Qing Wang, Jiechao Xiong, Lei Han, Meng Fang, Xinghai Sun, Zhuobin Zheng, Peng Sun, Zhengyou Zhang

专题命中 Agent评测 ：agent(title,abstract)；multi-agent(title,abstract)；分类 cs.AI、cs.LG

Comments 21 pages, 14 figures

Power Systems Agent Benchmark: Executable Evaluation of AI Agents in Electric Power Engineering

电力系统智能体基准测试：电力工程中AI智能体的可执行评估

Sergei Trashchenkov

机构 * Electric Power Engineering（电力工程） 纠错

专题命中 Agent评测 ：agent(title,abstract)；AI agent(title,abstract)；分类 cs.AI

AI总结 提出电力系统智能体基准测试，通过确定性评估器对智能体在电力工程任务中的结构化解决方案进行可执行评估，涵盖41个任务族，并采用私有种子按需生成保留案例以防止数据污染。

Comments 19 pages, 1 figure, 2 tables. Code and data: https://github.com/trashchenkov/power-systems-agent-benchmark ; archived at https://doi.org/10.5281/zenodo.20753046 v2: fixed unresolved citations and three missing references in Section 2, reference capitalization, Table 1 caption

Autodata: An agentic data scientist to create high quality synthetic data

Autodata: 一种创建高质量合成数据的智能数据科学家方法

Ilia Kulikov, Chenxi Whitehouse, Tianhao Wu, Yixin Nie, Swarnadeep Saha, Eryk Helenowski, Weizhe Yuan, Olga Golovneva, Jack Lanchantin, Yoram Bachrach, Jakob Foerster, Xian Li, Han Fang, Sainbayar Sukhbaatar, Jason Weston

机构 * Meta 纠错

专题命中 Agent评测 ：agentic(title,abstract)；agent(abstract)；AI agent(abstract)；分类 cs.AI、cs.CL、cs.LG

AI总结 提出Autodata方法，让AI代理充当数据科学家，通过元优化学习创建更优的训练和评估数据，在计算机科学、法律推理和数学推理任务上优于传统方法。

HeuriGym: An Agentic Benchmark for LLM-Crafted Heuristics in Combinatorial Optimization

HeuriGym: 一个用于评估LLM生成启发式算法的代理基准

Hongzheng Chen, Yingheng Wang, Yaohui Cai, Hins Hu, Jiajie Li, Shirley Huang, Chenhui Deng, Rongjian Liang, Shufeng Kong, Haoxing Ren, Samitha Samaranayake, Carla P. Gomes, Zhiru Zhang

专题命中 Agent评测 ：agentic(title,abstract)；agent(abstract)；tool use(abstract)；planning(abstract)

AI总结 HeuriGym是一个用于评估LLM生成启发式算法的代理基准，通过代码执行反馈和迭代改进，揭示了LLM在组合优化中的局限性。

Comments Accepted to ICLR'26

RocqStar: Leveraging Similarity-driven Retrieval and Agentic Systems for Rocq generation

RocqStar: 利用相似性驱动检索与智能体系统进行Rocq生成

Andrei Kozyrev, Nikita Khramov, Gleb Solovev, Anton Podkopaev

机构 * JetBrains Research（JetBrains研究） 纠错

专题命中 Agent评测 ：agentic(title,abstract)；agent(abstract)；planning(abstract)；multi-agent(abstract)

AI总结 RocqStar通过结合相似性驱动检索和智能体系统，显著提升Rocq证明生成的性能和稳定性。

Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures

因果智能体回放：LLM智能体故障的反事实归因

Jaineet Shah

机构 * Carnegie Mellon University（卡内基梅隆大学） 纠错

专题命中 Agent评测 ：agent(title,summary_cn)；分类 cs.AI、cs.LG

AI总结 提出Causal Agent Replay (CAR)方法，通过结构因果模型和干预操作，对LLM智能体失败步骤进行反事实归因，解决现有方法无法定位决策步骤的问题。

Comments Open-source: https://github.com/jaineet17/causal-agent-replay

BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics

BioAgent Bench: 一个用于生物信息学的AI代理评估套件

Dionizije Fa, Marko Culjak, Bruno Pandza, Mateo Cupic

机构 * Entropic 纠错

专题命中 Agent评测 ：agent(title,abstract)；AI agent(title,abstract)；分类 cs.AI

AI总结 本文提出BioAgent Bench，用于评估AI代理在常见生物信息学任务中的性能和鲁棒性。通过定制端到端任务和压力测试，发现前沿代理可完成多步骤流程，但鲁棒性测试揭示了在受控扰动下的失败模式，表明高阶流程构建不保证可靠步骤推理。

Comments ICML 2026 camera ready

402Pilot: An x402 Decision Layer for Autonomous Agent Micropayments

402Pilot：面向自主智能体微支付的x402决策层

Yin Li, Yanbo He, Boo-Ho Yang, Rav Lawana, Ziyue Li, Wei Zeng, Jing Tang, Fugee Tsung

专题命中 Agent评测 ：agent(title,abstract)；autonomous agent(title,abstract)；分类 cs.AI

AI总结 针对自主智能体微支付的买方决策问题，提出协议无关的402Pilot决策层，结合PA-DCT策略在402Pilot-Bench基准中验证其自适应采购的有效性，为可编程支付补充买方决策能力。

A Diagnostic Framework for AI Agent Behavior

人工智能代理行为的诊断框架

Xichen Zhang, Yingjie Zhang, Tianshu Sun

机构 * Cheung Kong Graduate School of Business（长江商学院） 纠错

专题命中 Agent评测 ：agent(title,abstract)；AI agent(title,abstract)；分类 cs.AI

AI总结 研究人工智能代理在复杂系统中的行为评估问题，提出层归因诊断框架，该框架含基础计算层和行为调制层，阐明了替代有效性等三个结果，为评估代理行为提供了新方法。

TrustX Agent Risk Classification Framework (ARC): Risk-Tiering Internally Created Agentic AI Systems

TrustX智能体风险分类框架（ARC）：对内部创建的智能体人工智能系统进行风险分层

Hannah M. Liu, Rhea Saxena, Shiv Asthana

专题命中 Agent评测 ：agent(title,abstract)；agentic(title,abstract)；分类 cs.AI

AI总结 针对智能体人工智能系统超出通用风险框架治理能力的问题，提出TrustX智能体风险分类框架（ARC），利用十二维度评分标准等组件量化风险，输出三层治理结果，为相关人员提供工具，且会持续迭代完善。

Comments This is a working paper on our risk classification tool, with iterations currently underway

Can Trustless Agents Be Trusted? An Empirical Study of the ERC-8004 Decentralized AI Agent Ecosystem

无信任代理能否被信任？ERC-8004 去中心化 AI 代理生态系统的实证研究

Xihan Xiong, Zelin Li, Wei Wei, Qin Wang, William Knottenbelt, Zhipeng Wang

机构 * Imperial College London（伦敦帝国学院） ； Ohio State University（俄亥俄州立大学） ； University of Bristol（布里斯托大学） ； The University of Manchester（曼彻斯特大学） 纠错

专题命中 Agent评测 ：agent(title,abstract)；AI agent(title,abstract)；分类 cs.AI

AI总结 本研究首次实证分析 ERC-8004 协议在三条链上的身份、声誉和验证注册表，发现大多数注册为占位符，声誉不可比且易被操纵，大量评论者存在 Sybil 行为，表明该协议目前无法提供可信决策基础。

ContextNest: Verifiable Context Governance for Autonomous AI Agent

ContextNest：自主AI智能体的可验证上下文治理

Misha Sulpovar, Benn R. Konsynski, Qaish Kanchwala, Gabe Goodhart

机构 * PromptOwl, LLC（PromptOwl公司） ； Goizueta Business School, Emory University（埃默里大学戈伊苏埃塔商学院） ； IBM Research（IBM研究院） 纠错

专题命中 Agent评测 ：agent(title,abstract)；AI agent(title,abstract)；分类 cs.AI

AI总结 提出ContextNest，一种在检索前提供上下文治理的开放规范，通过版本链、哈希校验和审计追踪确保知识可验证性，实验表明其在防过时攻击和检索确定性上优于传统方法。

Comments 35 pages, 11 tables, 4 figures

Behavioral Integrity Verification for AI Agent Skills

AI代理技能的行为完整性验证

Yuhao Wu, Tung-Ling Li, Hongliang Liu

机构 * Palo Alto Networks（帕洛阿尔托网络） 纠错

专题命中 Agent评测 ：agent(title,abstract)；AI agent(title)；分类 cs.AI

AI总结 本文提出行为完整性验证(BIV)方法，通过类型集比较验证AI代理技能声明与实际能力的一致性，发现80%的技能存在描述与实现差距，并在恶意技能检测中取得高F1值。