APort Vault：使用开放智能体护照基准测试AI智能体支付授权
APort Vault: Benchmarking AI Agent Payment Authorization with the Open Agent Passport
查看机构详情
- APort Technologies Inc.（APort科技有限公司）
机构由 AI 辅助整理，请以论文原文为准。
浏览论文内容
中文总结 AI 辅助
APort Vault基准测试通过重放4,371次攻击，评估AI智能体支付授权，验证开放智能体护照（OAP）层可将未经授权的转账降至零，同时保持高支付执行率。
中文摘要 AI 辅助
APort Vault是一个用于工具使用型AI智能体支付授权的基准测试。它重放了在公开夺旗赛期间人类针对实时支付智能体编写的4,371次攻击，覆盖来自8个实验室的14个模型、五种策略配置和两条重放轨道，分别在有和没有实现开放智能体护照（OAP）规范的确定性预操作检查的情况下进行。共完成了225,964次评估。我们每次评估报告五个不同事件，因为将它们合并正是智能体基准测试产生无法通过审查的数字的方式。请求很常见，其速率在不同配置间的差异远大于不同模型间的差异，尽管每次攻击恰好存在于一种配置中，因此策略和攻击队列共同变化：仅模型评估中10.9%处于级别1，3.0%处于级别2，0.1%处于级别3，79.4%处于级别4。在1,293个级别4提示上，每个提示在所有模型上评估，请求率从71.2%到84.3%不等，其中809个提示（62.6%）从全部十四个模型引发了请求，每个请求最终都向该级别白名单中的收款人成功付款。授权边界是条件出现分歧的地方。在级别2到4，向护照不允许的收款人的转账在仅模型情况下为76,842次中的140次，而在该层之后为69,297次中的0次，并且在68,970个匹配的模型、提示和轨道三元组上为105次对0次。零跨越790个源会话，给出每个会话的上限为0.38%。这不是通过拒绝付款实现的：在该层之后执行了25,370笔付款，而策略拒绝了其评估的25,640次转账调用中的187次，其中148次是因为收款人被禁止。我们在以下网址发布225,964次评估、级别护照、评分代码和分析脚本：此HTTP URL。
英文摘要
APort Vault is a benchmark for payment authorization in tool-using AI agents. It replays 4,371 attacks written by humans against a live payment agent during a public capture-the-flag event, across 14 models from 8 labs, five policy configurations and two replay tracks, with and without a deterministic pre-action check implementing the Open Agent Passport (OAP) specification. 225,964 evaluations completed. We report five distinct events per evaluation, because collapsing them is how an agent benchmark produces a number that does not survive review. Requests are common and their rate differs far more across configurations than across models, though each attack exists at exactly one configuration so policy and attack cohort vary together: 10.9% of model-alone evaluations at Level 1, 3.0% at Level 2, 0.1% at Level 3, 79.4% at Level 4. On the 1,293 Level 4 prompts, each evaluated on every model, request rates run from 71.2% to 84.3%, and 809 prompts (62.6%) elicited a request from all fourteen models, each ending in a successful payment to the level's allowlisted recipient. The authorization boundary is where the conditions diverge. At Levels 2 to 4, transfers to recipients the passport did not permit number 140 of 76,842 with the model alone and 0 of 69,297 behind the layer, and 105 against 0 on 68,970 matched model, prompt and track triples. The zero spans 790 source sessions, giving a per-session upper bound of 0.38%. It was not obtained by refusing payments: 25,370 payments executed behind the layer, while the policy denied 187 of the 25,640 transfer calls it evaluated, 148 of them for a forbidden recipient. We release the 225,964 evaluations, the level passports, the scoring code and the analysis script at huggingface.co/datasets/aporthq/vault-benchmark-v1 .
