arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.13239cs.AI
收藏

对Solidity智能合约错误检测零样本推理方法的基准测试

Benchmarking Zero-Shot Reasoning Approaches for Error Detection in Solidity Smart Contracts

TeleMídia Lab - PUC-Rio(TeleMídia实验室-巴西利亚大学);BTG Pactual;Instituto Militar de Engenharia(军事工程学院)

Eduardo Sardenberg, Antonio José Grandson Busson, Daniel de Sousa Moraes, Julio Cesar Duarte, Sérgio Colcher

AI总结:

本文评估了最新LLM在Solidity智能合约分析中的表现,通过400个合约数据集,在错误检测和分类任务中比较零样本提示策略的效果,发现CoT和ToT显著提升召回率但降低精度,Claude 3 Opus在ToT提示下取得最佳F1分数。

摘要:

智能合约在区块链系统中扮演核心角色,通过编码金融和运营逻辑。然而,其对细微安全漏洞的易受攻击性使财务损失和信任侵蚀风险显著。LLMs为自动化漏洞检测创造了新机会,但不同提示策略和模型选择在现实中的有效性仍不确定。本文使用400个合约的平衡数据集,评估了最先进的LLM在Solidity智能合约分析中的表现,通过两个任务:(i) 错误检测,模型进行二分类以决定合约是否易受攻击;(ii) 错误分类,模型必须将预测问题分配到特定漏洞类别。模型使用零样本提示策略进行评估,包括零样本、零样本链式思维(CoT)和零样本树状思维(ToT)。在错误检测任务中,CoT和ToT显著增加召回率(通常接近~95-99%),但通常降低精度,表明更敏感的决策模式,带来更多假阳性。在错误分类任务中,Claude 3 Opus在ToT提示下取得最佳加权F1分数(90.8),随后是其CoT。

英文摘要:

Smart contracts play a central role in blockchain systems by encoding financial and operational logic. Still, their susceptibility to subtle security flaws poses significant risks of financial loss and erosion of trust. LLMs create new opportunities for automating vulnerability detection, yet the effectiveness of different prompting strategies and model choices in real-world contexts remains uncertain. This paper evaluates state-of-the-art LLMs on Solidity smart contract analysis using a balanced dataset of 400 contracts under two tasks: (i) Error Detection, where the model performs binary classification to decide whether a contract is vulnerable, and (ii) Error Classification, where the model must assign the predicted issue to a specific vulnerability category. Models are evaluated using zero-shot prompting strategies, including zero-shot, zero-shot Chain-of-Thought (CoT), and zero-shot Tree-of-Thought (ToT). In the Error Detection task, CoT and ToT substantially increase recall (often approaching ~ 95--99%), but typically reduce precision, indicating a more sensitive decision regime with more false positives. In the Error Classification task, Claude 3 Opus attains the best Weighted F1-score (90.8) under the ToT prompt, followed closely by its CoT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑