OpenPerf：面向开源生态系统可持续发展的基准测试框架
OpenPerf: A Benchmarking Framework for the Sustainable Development of the Open-Source Ecosystem
AI总结：
本文提出OpenPerf基准测试框架，针对开源生态系统可持续发展，定义9项任务涵盖3种数据类型解决6个研究问题，实现6个基准并提供BaaS工具包，已被多家知名机构应用验证。
摘要：
基准测试涉及设计科学的测试方法、工具和框架，以定量且可比较地评估特定测试对象的特定性能指标。随着人工智能的发展，ImageNet和DataPerf等AI基准测试数据集已逐渐成为学术界和工业界的共识标准。然而，由于数据类型的多样性、研究问题的广泛性以及协作网络的复杂性，在开源领域构建基准测试框架仍然是一个重大挑战。本文介绍了OpenPerf，一个为开源生态系统可持续发展设计的基准测试框架。该框架定义了开源研究中的9项基准测试任务，涵盖3种数据类型：时间序列、文本和图形，并解决6个研究问题，包括回归、分类、推荐、排序、网络构建和异常检测。基于上述任务，我们实现了3个数据科学任务基准、2个基于索引的基准和1个标准基准。值得注意的是，基于索引的基准已被中国电子技术标准化研究院采用，作为开源社区治理的评估标准。此外，我们为OpenPerf开发了全面的工具包，不仅提供强大的数据管理、工具集成和用户界面功能，还采用基准测试即服务模型为学术机构、行业和基金会提供服务。通过在阿里巴巴、蚂蚁集团和华东师范大学等知名企业和机构的应用，我们验证了OpenPerf在开源生态系统健康发展中的关键作用。
英文摘要：
Benchmarking involves designing scientific test methods, tools, and frameworks to quantitatively and comparably assess specific performance indicators of certain test subjects. With the development of artificial intelligence, AI benchmarking datasets such as ImageNet and DataPerf have gradually become consensus standards in both academic and industrial fields. However, constructing a benchmarking framework remains a significant challenge in the open-source domain due to the diverse range of data types, the wide array of research issues, and the intricate nature of collaboration networks. This paper introduces OpenPerf, a benchmarking framework designed for the sustainable development of the open-source ecosystem. This framework defines 9 task benchmarking tasks in the open-source research, encompassing 3 data types: time series, text, and graphics, and addresses 6 research problems including regression, classification, recommendation, ranking, network building, and anomaly detection. Based on the above tasks, we implemented 3 data science task benchmarks, 2 index-based benchmarks, and 1 standard benchmark. Notably, the index-based benchmarks have been adopted by the China Electronics Standardization Institute as evaluation criteria for open-source community governance. Additionally, we have developed a comprehensive toolkit for OpenPerf, which not only offers robust data management, tool integration, and user interface capabilities but also adopts a Benchmarking-as-a-Service (BaaS) model to serve academic institutions, industries, and foundations. Through its application in renowned companies and institutions such as Alibaba, Ant Group, and East China Normal University, we have validated OpenPerf's pivotal role in the healthy evolution of the open-source ecosystem.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
