Existing evaluation frameworks for large language models -- including HELM, MT-Bench, AgentBench, and BIG-bench -- are designed for controlled, single-session, lab-scale settings. They do not address the evaluation challenges that emerge when agentic AI systems operate continuously in production: compounding decision errors, tool failure cascades, non-deterministic output drift, and the absence of ground truth for long-horizon tasks. This paper makes three contributions. First, we present a taxonomy of seven failure modes unique to production agentic systems, each grounded in observations from systems operating at billion-event scale. Second, we demonstrate empirically where standard metrics -- ROUGE, BERTScore, accuracy/AUC, and the agentic benchmarks above -- fail to detect each failure mode. Third, we propose PAEF (Production Agentic Evaluation Framework), a five-dimension evaluation framework with an open-source reference implementation, designed for continuous evaluation on production traffic rather than episodic benchmark runs. Our analysis shows that standard metrics fail to detect four of the seven failure modes entirely and detect three others only after a lag of multiple evaluation cycles.
Governance-Constrained Agentic AI: Blockchain-Enforced Human Oversight for Safety-Critical Wildfire Monitoring
AI Center, Faculty of Computer and Information Systems, Islamic University of Madinah(伊斯兰大学麦地那分校计算机与信息系统学院人工智能中心);Faculty of Computer Studies, Arab Open University-Bahrain(阿拉伯开放大学巴林分校计算机研究学院);Department of Computer Science, The Islamia University of Bahawalpur(巴哈瓦尔布尔伊斯兰大学计算机科学系);College of Computer Studies, University of Technology Bahrain(巴林科技大学计算机研究学院)
RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training
Adelaide University(阿德莱德大学);The University of Sydney(悉尼大学);Curtin University(科廷大学);School of CSIT(计算机科学与信息技术学院);ACFR(澳大利亚空间研究中心);School of EECMS(电子工程、计算机与数学科学学院)
Beyond the 'Diff': Addressing Agentic Entropy in Agentic Software Development
Dipartimento Tecnologie Innovative, SUPSI(技术创新部,SUPSI);Dalle Molle Institute for Artificial Intelligence (IDSIA), SUPSI(达勒莫利人工智能研究所(IDSIA),SUPSI);Management in Networked and Digital Societies (MINDS) Department, Kozminski University(网络化与数字化社会管理(MINDS)系,科津斯基大学);Institute of Biomedical Ethics and History of Medicine, University of Zurich(生物医学伦理与医学史研究所,苏黎世大学);ETH Zurich(苏黎世联邦理工学院)
Agentic Peer-to-Peer Networks: From Content Distribution to Capability and Action Sharing
College of Electronics and Information Engineering, Shenzhen University(深圳大学电子与信息工程学院);School of Informatics, Xiamen University(厦门大学信息学院);School of Computer Science and Cyber Engineering, Guangzhou University(广州大学计算机科学与网络工程学院)
From Proprietary to Open-Source: Bridging the Distribution Gap via Multi-Agent Protocol Distillation in Agentic Search
Beijing Institute of Technology(北京理工大学);East China Normal University(华东师范大学);University of Science and Technology of China(中国科学技术大学);Tsinghua University(清华大学);University of Chinese Academy of Sciences(中国科学院大学)
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));DeepWisdom;Singapore University of Technology and Design(新加坡科技设计大学);Nanyang Technological University(南洋理工大学);Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学);Université de Montréal & Mila(蒙特利尔大学及Mila)
PhotoCraft: Agentic Reasoning with Hierarchical Self-Evolving Memory for Deep Image Search
Pattern Recognition Center, WeChat AI, Tencent Inc.(微信AI模式识别中心,腾讯公司);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Zhongguancun Academy(中关村学院);Nanyang Technological University(南洋理工大学);Shanghai Jiao Tong University(上海交通大学)