DMind Benchmark：面向Web3领域LLM能力的全面评估
DMind Benchmark: Toward a Holistic Assessment of LLM Capabilities across the Web3 Domain
AI总结：
提出DMind Benchmark，覆盖Web3九个子领域，结合客观知识与开放推理任务，评估31个LLM，发现模型在安全审计等高推理任务中存在显著弱点。
摘要：
Web3生态系统以密码学原语和去中心化共识为基础，是一个高风险环境，软件漏洞和激励错配直接导致财务损失。随着大语言模型（LLM）越来越多地集成到该领域，用于智能合约审计、去中心化金融分析等任务，确保其可靠性至关重要。然而，通用基准无法捕捉这些对抗性和协议驱动环境所需的专门推理。为弥补这一差距，我们引入DMind Benchmark，这是一个全面的评估套件，旨在严格评估LLM在Web3栈中的能力。DMind Benchmark涵盖九个不同的子领域（涵盖基础设施、智能合约、代币经济学等），并将客观知识检索与模拟真实世界操作挑战的复杂开放式推理任务相结合。我们对31个领先的专有和开源权重模型进行了广泛评估，采用污染感知管道，并通过严格的跨评判一致性检查验证评分协议的统计稳健性。我们的分析揭示了一个关键二分法：虽然模型在基础基础设施概念上表现出能力，但在安全审计等高推理任务中表现出显著弱点。此外，我们提供了帕累托分析以指导成本效益部署，并通过对抗性实验证明，在DMind Benchmark上取得高性能需要真正的推理而非表面记忆。自2025年4月开源发布以来，DMind Benchmark在Hugging Face上连续近一周位居趋势第一，到2026年6月累计下载量超过13k，成为Web3中推进安全可信AI的标准。
英文摘要：
The Web3 ecosystem, underpinned by cryptographic primitives and decentralized consensus, represents a high-stakes environment where software vulnerabilities and incentive misalignments translate directly into financial loss. As Large Language Models (LLMs) are increasingly integrated into this domain for tasks ranging from smart contract auditing to decentralized finance analytics, ensuring their reliability is paramount. However, general-purpose benchmarks fail to capture the specialized reasoning required for these adversarial and protocol-driven settings. To bridge this gap, we introduce DMind Benchmark, a comprehensive evaluation suite designed to rigorously assess LLM proficiency across the Web3 stack. DMind Benchmark encompasses nine distinct subdomains (spanning infrastructure, smart contracts, token economics, etc.) and combines objective knowledge retrieval with complex open-ended reasoning tasks that emulate real-world operational challenges. We conduct an extensive evaluation of 31 leading proprietary and open-weights models, employing a contamination-aware pipeline and verifying the statistical robustness of our scoring protocol through rigorous cross-judge consistency checks. Our analysis reveals a critical dichotomy: while models demonstrate competence in foundational infrastructure concepts, they exhibit significant vulnerabilities in high-reasoning tasks such as security auditing. Furthermore, we provide a Pareto analysis to guide cost-effective deployment and demonstrate through adversarial experiments that high performance on DMind Benchmark necessitates genuine reasoning rather than superficial memorization. Since its open-source release in April 2025, DMind Benchmark achieved the #1 trending position on Hugging Face for nearly a week and accumulated over 13k downloads by June 2026, establishing itself as a standard for advancing secure and trustworthy AI in Web3.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
