SWE-AGI：在自主代理时代利用MoonBit进行规范驱动软件构建的基准测试
SWE-AGI: Benchmarking Specification-Driven Software Construction with MoonBit in the Era of Autonomous Agents
AI总结：
SWE-AGI通过MoonBit评估LLM在规范驱动下的软件构建能力，揭示了AI辅助开发中代码阅读瓶颈及模型性能随任务难度变化的特性。
摘要：
尽管大型语言模型（LLMs）已展示出令人印象深刻的编程能力，但它们能否从明确的规范中自主构建生产级软件仍是一个开放性问题。我们介绍了SWE-AGI，一个开源基准，用于评估用MoonBit编写的软件系统的端到端、规范驱动的构建。SWE-AGI任务要求基于标准和RFCs的LLM代理严格实现解析器、解释器、二进制解码器和SAT求解器，其在固定API框架下。每个任务涉及实现1,000-10,000行的核心逻辑，相当于有经验的开发人员数周或数月的工作量。通过利用新兴的MoonBit生态系统，SWE-AGI减少了数据泄露，迫使代理依赖于长周期的架构推理，而不是代码检索。在前沿模型中，gpt-5.3-codex在总体表现上最佳（解决19/22个任务，86.4%），优于claude-opus-4.6（15/22，68.2%），而kimi-2.5在开源模型中表现最强。随着任务难度增加，性能急剧下降，特别是在规范密集型系统上。行为分析进一步揭示，随着代码库扩大，代码阅读而非编写成为AI辅助开发的主要瓶颈。总体而言，虽然规范驱动的自主软件工程日益可行，但在可靠支持生产级开发之前仍存在重大挑战。
英文摘要：
Although large language models (LLMs) have demonstrated impressive coding capabilities, their ability to autonomously build production-scale software from explicit specifications remains an open question. We introduce SWE-AGI, an open-source benchmark for evaluating end-to-end, specification-driven construction of software systems written in MoonBit. SWE-AGI tasks require LLM-based agents to implement parsers, interpreters, binary decoders, and SAT solvers strictly from authoritative standards and RFCs under a fixed API scaffold. Each task involves implementing 1,000-10,000 lines of core logic, corresponding to weeks or months of engineering effort for an experienced human developer. By leveraging the nascent MoonBit ecosystem, SWE-AGI minimizes data leakage, forcing agents to rely on long-horizon architectural reasoning rather than code retrieval. Across frontier models, gpt-5.3-codex achieves the best overall performance (solving 19/22 tasks, 86.4%), outperforming claude-opus-4.6 (15/22, 68.2%), and kimi-2.5 exhibits the strongest performance among open-source models. Performance degrades sharply with increasing task difficulty, particularly on hard, specification-intensive systems. Behavioral analysis further reveals that as codebases scale, code reading, rather than writing, becomes the dominant bottleneck in AI-assisted development. Overall, while specification-driven autonomous software engineering is increasingly viable, substantial challenges remain before it can reliably support production-scale development.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
