ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions
Aayush Gupta
备注:
18 pages, 5 figures, 8 tables. Evaluates ReAct vs Reflexion across four tool-using domains with perturbation (epsilon) and fault-injection (lambda) stress testing; 1,280 total episodes
Existing benchmarks for tool-using LLM agents primarily report single-run success rates and miss reliability properties required in production. We introduce \textbf{ReliabilityBench}, a benchmark for evaluating agent reliability across three dimensions: (i) consistency under repeated execution using $\mathrm{pass}^k$, (ii) robustness to semantically equivalent task perturbations at intensity $ε$, and (iii) fault tolerance under controlled tool/API failures at intensity $λ$. ReliabilityBench contributes a unified reliability surface $R(k,ε,λ)$, \textit{action metamorphic relations} that define correctness via end-state equivalence rather than text similarity, and a chaos-engineering-style fault injection framework (timeouts, rate limits, partial responses, schema drift). We evaluate two models (Gemini 2.0 Flash, GPT-4o) and two agent architectures (ReAct, Reflexion) across four domains (scheduling, travel, customer support, e-commerce) over 1,280 episodes. Perturbations alone reduce success from 96.9% at $ε=0$ to 88.1% at $ε=0.2$. Rate limiting is the most damaging fault in ablations. ReAct is more robust than Reflexion under combined stress, and Gemini 2.0 Flash achieves comparable reliability to GPT-4o at much lower cost. ReliabilityBench provides a systematic framework for assessing production readiness of LLM agents.
FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions
Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学);Carnegie Mellon University(卡内基梅隆大学);Seoul National University(首尔大学)
Marking the Wrong Symptoms: Evaluating LLM Watermarks in Medical Texts
ETH Zurich(苏黎世联邦理工学院);Berlin Institute of Health at Charité – Universitätsmedizin Berlin(柏林夏里特医学院柏林健康研究所);Deutsches Herzzentrum der Charité – Medical Heart Center of Charité and German Heart Institute Berlin(柏林夏里特医学院德国心脏中心与柏林德国心脏研究所)
SVC-Probe: A Framework for Evaluating Perturbation Generalization in Spatial Foundation-Model Embeddings
Department of Biomedical Informatics and Data Science, University of Alabama at Birmingham(亚拉巴马大学伯明翰分校生物医学信息学与数据科学系);System Pharmacology and AI Research Center, University of Alabama at Birmingham(亚拉巴马大学伯明翰分校系统药理学与人工智能研究中心)