DEP:一种去中心化的大型语言模型评估协议
DEP: A Decentralized Large Language Model Evaluation Protocol
AI总结:
DEP提出一种去中心化的评估协议,通过匹配服务器实现统一、标准化的LLM评估,支持模块化和即插即用的评估方式,减少基准测试泄露风险并提高可重复性。
摘要:
随着大型语言模型(LLMs)的快速发展,已经提出了大量基准测试。然而,大多数基准测试缺乏统一的评估标准,且需要手动实现定制脚本,导致结果难以保证一致性和可重复性。此外,主流评估框架是集中式的,包含数据集和答案,这增加了基准测试泄露的风险。为了解决这些问题,我们提出了一种去中心化的评估协议(DEP),通过匹配服务器实现一种去中心化但统一和标准化的评估框架,而无需限制基准测试。服务器可以本地安装或远程部署,一旦适配,就可以长期重复使用。通过将用户、LLMs和基准测试解耦,DEP实现了模块化、即插即用的评估:基准测试文件和评估逻辑始终保持在服务器一侧。在远程设置中,用户无法访问真实数据,从而实现数据隔离和防泄漏评估。为了便于实际应用,我们开发了DEP工具包,一个兼容协议的工具包,支持断点续传、并发请求和拥堵控制等功能。我们还提供了详细的文档,帮助适配新的基准测试到DEP。使用DEP工具包,我们对多个LLM进行了跨基准测试的评估。实验结果验证了DEP的有效性,并表明它减少了部署基准测试评估的成本。截至2026年2月,我们已适配超过60个基准测试,并继续推动社区共建,以支持跨各种任务和领域的统一评估。
英文摘要:
With the rapid development of Large Language Models (LLMs), a large number of benchmarks have been proposed. However, most benchmarks lack unified evaluation standard and require the manual implementation of custom scripts, making results hard to ensure consistency and reproducibility. Furthermore, mainstream evaluation frameworks are centralized, with datasets and answers, which increases the risk of benchmark leakage. To address these issues, we propose a Decentralized Evaluation Protocol (DEP), a decentralized yet unified and standardized evaluation framework through a matching server without constraining benchmarks. The server can be mounted locally or deployed remotely, and once adapted, it can be reused over the long term. By decoupling users, LLMs, and benchmarks, DEP enables modular, plug-and-play evaluation: benchmark files and evaluation logic stay exclusively on the server side. In remote setting, users cannot access the ground truth, thereby achieving data isolation and leak-proof evaluation. To facilitate practical adoption, we develop DEP Toolkit, a protocol-compatible toolkit that supports features such as breakpoint resume, concurrent requests, and congestion control. We also provide detailed documentation for adapting new benchmarks to DEP. Using DEP toolkit, we evaluate multiple LLMs across benchmarks. Experimental results verify the effectiveness of DEP and show that it reduces the cost of deploying benchmark evaluations. As of February 2026, we have adapted over 60 benchmarks and continue to promote community co-construction to support unified evaluation across various tasks and domains.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。