迈向自动智能合约生成：评估、基准测试与检索增强修复
Towards Automated Smart Contract Generation: Evaluation, Benchmarking, and Retrieval-Augmented Repair
AI总结：
本文提出SolBench和RAR框架，通过基于执行的基准测试和检索增强修复方法，提升Solidity智能合约生成的准确性和效率。
摘要：
智能合约，主要用Solidity编写并部署在以太坊等区块链上，部署后不可更改，因此功能正确性至关重要。然而，现有的Solidity代码生成评估主要依赖于表面级指标（例如BLEU、CrystalBLEU）或人工检查，这些指标与功能正确性相关性较差。与Python不同，Solidity缺乏大规模、基于执行的基准测试，限制了对大型语言模型在智能合约开发中的系统评估。我们介绍了SolBench，一个全面的基准测试和自动化测试流水线，用于Solidity，通过差分模糊测试强调功能正确性。SolBench包含从Etherscan（genesis-2024）收集的7604个真实智能合约中提取的28825个函数，涵盖十个应用领域。我们基准测试了14种不同的LLM，涵盖开源和闭源模型、13亿至671亿参数，以及通用和代码专用架构。主要的失败模式是缺少关键的合约内部信息，如状态变量和类型定义。提供完整的合约上下文可以提高准确性，但会带来可接受的推理成本。为了解决这个问题，我们提出了检索增强修复（RAR），一种成本效益高的框架，将执行反馈整合到代码修复中。RAR使用编译器和运行时错误信息来检索仅需纠正目标函数的最小合约片段，避免完整上下文推理。这显著减少了输入长度同时提高了功能正确性。我们进一步分析了RAR中的检索和修复策略，展示了在准确性和效率上的持续收益。SolBench和RAR使基于执行的评估和经济的改进成为可能。数据集和代码可在https://github.com/ZaoyuChen/SolBench上公开获取。
英文摘要：
Smart contracts, predominantly written in Solidity and deployed on blockchains such as Ethereum, are immutable after deployment, making functional correctness critical. However, existing evaluations of Solidity code generation rely largely on surface-level metrics (e.g., BLEU, CrystalBLEU) or manual inspection, which correlate poorly with functional correctness. In contrast to Python, Solidity lacks large-scale, execution-based benchmarks, limiting systematic evaluation of large language models for smart contract development. We introduce SolBench, a comprehensive benchmark and automated testing pipeline for Solidity that emphasizes functional correctness via differential fuzzing. SolBench consists of 28825 functions extracted from 7604 real-world smart contracts collected from Etherscan (genesis-2024), spanning ten application domains. We benchmark 14 diverse LLMs, covering open and closed models, 1.3B-671B parameters, and both general-purpose and code-specialized architectures. The dominant failure mode is missing critical intra-contract information, such as state variables and type definitions. Providing full-contract context improves accuracy but incurs prohibitive inference costs. To address this, we propose Retrieval-Augmented Repair (RAR), a cost-effective framework that integrates execution feedback into code repair. RAR uses compiler and runtime error messages to retrieve only the minimal contract snippets needed to correct a target function, avoiding full-context inference. This significantly reduces input length while improving functional correctness. We further analyze retrieval and repair strategies within RAR, demonstrating consistent gains in accuracy and efficiency. SolBench and RAR enable principled, execution-based evaluation and economical improvement of Solidity code generation. Dataset and code are publicly available at https://github.com/ZaoyuChen/SolBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
