arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.06112cs.AI
收藏

ReliabilityBench: 评估LLM代理在生产类压力条件下可靠性

ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions

Aayush Gupta

AI总结:

ReliabilityBench通过评估LLM代理在生产类压力条件下的可靠性,提出统一的可靠性表面和动作元关系,评估不同模型和架构的可靠性表现。

摘要:

现有用于工具使用LLM代理的基准测试主要报告单次运行的成功率,忽略了生产中所需的可靠性属性。我们引入ReliabilityBench,一个用于评估代理可靠性三个维度的基准测试:(i) 使用pass^k进行重复执行的一致性,(ii) 在强度为ε的语义等价任务扰动下的鲁棒性,(iii) 在受控工具/API故障强度为λ下的容错性。ReliabilityBench贡献了一个统一的可靠性表面R(k,ε,λ),定义正确性的动作元关系通过终态等价而非文本相似性,以及一种类似混沌工程的故障注入框架(超时、速率限制、部分响应、模式漂移)。我们评估了两个模型(Gemini 2.0 Flash,GPT-4o)和两个代理架构(ReAct,Reflexion)在四个领域(调度、旅行、客户支持、电子商务)上的1280个回合。扰动单独使成功率从ε=0时的96.9%降至ε=0.2时的88.1%。速率限制是消融中最具破坏性的故障。ReAct在综合压力下比Reflexion更鲁棒,Gemini 2.0 Flash在成本远低于GPT-4o的情况下实现了与GPT-4o相当的可靠性。ReliabilityBench提供了一个系统框架,用于评估LLM代理的生产就绪性。

英文摘要:

Existing benchmarks for tool-using LLM agents primarily report single-run success rates and miss reliability properties required in production. We introduce \textbf{ReliabilityBench}, a benchmark for evaluating agent reliability across three dimensions: (i) consistency under repeated execution using $\mathrm{pass}^k$, (ii) robustness to semantically equivalent task perturbations at intensity $ε$, and (iii) fault tolerance under controlled tool/API failures at intensity $λ$. ReliabilityBench contributes a unified reliability surface $R(k,ε,λ)$, \textit{action metamorphic relations} that define correctness via end-state equivalence rather than text similarity, and a chaos-engineering-style fault injection framework (timeouts, rate limits, partial responses, schema drift). We evaluate two models (Gemini 2.0 Flash, GPT-4o) and two agent architectures (ReAct, Reflexion) across four domains (scheduling, travel, customer support, e-commerce) over 1,280 episodes. Perturbations alone reduce success from 96.9% at $ε=0$ to 88.1% at $ε=0.2$. Rate limiting is the most damaging fault in ablations. ReAct is more robust than Reflexion under combined stress, and Gemini 2.0 Flash achieves comparable reliability to GPT-4o at much lower cost. ReliabilityBench provides a systematic framework for assessing production readiness of LLM agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑