arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.10795cs.CRcs.ET
收藏

重新评估 EVMBench:AI代理是否准备好应对智能合约安全?

Re-Evaluating EVMBench: Are AI Agents Ready for Smart Contract Security?

Zhejiang University(浙江大学);BlockSec

Chaoyuan Peng, Lei Wu, Yajin Zhou

AI总结:

重新评估EVMBench发现AI代理在智能合约安全检测中存在稳定性、真实事件处理和框架依赖性问题,挑战了自动化审计的可行性。

摘要:

EVMbench由OpenAI、Paradigm和OtterSec发布,是首个大规模的AI代理在智能合约安全领域的基准测试。其结果——代理能够检测出多达45.6%的漏洞并利用72.2%的精选子集——激发了人们对完全自动化AI审计即将实现的期望。我们发现有两个局限性:其评估范围狭窄(14个代理配置,大多数模型仅在供应商提供的框架上测试)以及其依赖于在每个模型发布前发布的审计竞赛数据,这些数据模型可能在训练过程中已见过。为了解决这些问题,我们扩展到26个配置,涵盖四个模型家族和三个框架,并引入了一个无污染的数据集,包含22个真实世界的安全事件,这些事件发生在每个模型发布日期之后。我们的评估得出三个发现:(1)代理的检测结果不稳定,排名在不同配置、任务和数据集之间变化;(2)在真实世界事件中,没有任何代理能够在所有110个代理-事件对上完成端到端利用,尽管能够检测出多达65%的漏洞,这与EVMbench的结论相矛盾,即发现是主要瓶颈;(3)框架对结果有实质性影响,开源框架在某些方面优于供应商替代品,但EVMbench并未对此进行控制。这些发现挑战了完全自动化AI审计即将来临的叙事。代理能够可靠地捕捉已知模式并强烈响应人类提供的上下文,但无法取代人类判断。对于开发者,代理扫描可作为预部署检查。对于审计公司,代理在人类在场的工作流程中效果最佳,其中AI负责广度,而人类审计员贡献协议特定的知识和对抗性推理。代码和数据:https://github.com/blocksecteam/ReEVMBench/。

英文摘要:

EVMbench, released by OpenAI, Paradigm, and OtterSec, is the first large-scale benchmark for AI agents on smart contract security. Its results -- agents detect up to 45.6% of vulnerabilities and exploit 72.2% of a curated subset -- have fueled expectations that fully automated AI auditing is within reach. We identify two limitations: its narrow evaluation scope (14 agent configurations, most models tested on only their vendor scaffold) and its reliance on audit-contest data published before every model's release that models may have seen during training. To address these, we expand to 26 configurations across four model families and three scaffolds, and introduce a contamination-free dataset of 22 real-world security incidents postdating every model's release date. Our evaluation yields three findings: (1) agents' detection results are not stable, with rankings shifting across configurations, tasks, and datasets; (2) on real-world incidents, no agent succeeds at end-to-end exploitation across all 110 agent-incident pairs despite detecting up to 65% of vulnerabilities, contradicting EVMbench's conclusion that discovery is the primary bottleneck; and (3) scaffolding materially affects results, with an open-source scaffold outperforming vendor alternatives by up to 5 percentage points, yet EVMbench does not control for this. These findings challenge the narrative that fully automated AI auditing is imminent. Agents reliably catch well-known patterns and respond strongly to human-provided context, but cannot replace human judgment. For developers, agent scans serve as a pre-deployment check. For audit firms, agents are most effective within a human-in-the-loop workflow where AI handles breadth and human auditors contribute protocol-specific knowledge and adversarial reasoning. Code and data: https://github.com/blocksecteam/ReEVMBench/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑