arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.00822cs.DCcs.PF
收藏

CGSim:面向大规模分布式计算环境的仿真框架

CGSim: A Simulation Framework for Large Scale Distributed Computing Environment

Brookhaven National Laboratory(布鲁克海文国家实验室);University of Pittsburgh(匹兹堡大学);Carnegie Mellon University(卡内基梅隆大学);Oak Ridge National Laboratory(橡树岭国家实验室);University of Massachusetts(马萨诸塞大学);SLAC National Accelerator Laboratory(斯坦福直线加速器中心)
Sairam Sri Vatsavai, Raees Khan, Kuan-Chieh Hsu, Ozgur O. Kilic, Paul Nilsson, Tatiana Korchuganova, David K. Park, Sankha Dutta, Yihui Ren, Joseph Boudreau, Ta… 展开作者

Sairam Sri Vatsavai, Raees Khan, Kuan-Chieh Hsu, Ozgur O. Kilic, Paul Nilsson, Tatiana Korchuganova, David K. Park, Sankha Dutta, Yihui Ren, Joseph Boudreau, Tasnuva Chowdhury, Shengyu Feng, Jaehyung Kim, Scott Klasky, Tadashi Maeno, Verena Ingrid Martinez, Norbert Podhorszki, Frédéric Suter, Wei Yang, Yiming Yang, Shinjae Yoo, Alexei Klimentov, Adolfy Hoisie

AI总结:

针对现有分布式计算仿真器的局限,研究团队基于SimGrid构建了CGSim仿真框架,具备模块化插件、实时可视化、AI数据集生成等特性,经ATLAS PanDA工作负载验证,校准精度高且可实现近线性扩展,支持WLCG级规模仿真。

摘要:

大型强子对撞机全球计算网格(WLCG)等大规模分布式计算基础设施需要综合性仿真工具来评估性能、测试新算法并优化资源分配策略。然而,现有仿真器存在可扩展性有限、算法硬编码、缺乏实时监控、无法生成适配现代机器学习方法的数据集等问题。我们提出CGSim——一款面向大规模分布式计算环境的仿真框架,以解决上述局限。CGSim基于经过验证的SimGrid仿真框架构建,提供高层抽象用于建模异构网格环境,同时保持准确性与可扩展性。其核心特性包括:用于测试自定义工作流调度与数据传输策略的模块化插件机制、交互式实时可视化仪表盘、以及可自动生成适用于AI辅助性能建模的事件级数据集。我们通过使用生产环境中的ATLAS PanDA工作负载开展综合评估,验证了CGSim的能力,结果显示其在WLCG各计算站点的校准精度有显著提升。可扩展性实验表明,多站点仿真呈现近线性扩展特性,分布式工作负载的性能相比单站点执行提升6倍。该框架使研究人员能够在普通硬件上、在合理的时间预算内,仿真包含数百个站点、数千个并发作业的WLCG规模基础设施。

英文摘要:

Large-scale distributed computing infrastructures such as the Worldwide LHC Computing Grid (WLCG) require comprehensive simulation tools for evaluating performance, testing new algorithms, and optimizing resource allocation strategies. However, existing simulators suffer from limited scalability, hardwired algorithms, lack of real-time monitoring, and inability to generate datasets suitable for modern machine learning approaches. We present CGSim, a simulation framework for large-scale distributed computing environments that addresses these limitations. Built upon the validated SimGrid simulation framework, CGSim provides high-level abstractions for modeling heterogeneous grid environments while maintaining accuracy and scalability. Key features include a modular plugin mechanism for testing custom workflow scheduling and data movement policies, interactive real-time visualization dashboards, and automatic generation of event-level datasets suitable for AI-assisted performance modeling. We demonstrate CGSim's capabilities through a comprehensive evaluation using production ATLAS PanDA workloads, showing significant calibration accuracy improvements across WLCG computing sites. Scalability experiments show near-linear scaling for multi-site simulations, with distributed workloads achieving 6x better performance compared to single-site execution. The framework enables researchers to simulate WLCG-scale infrastructures with hundreds of sites and thousands of concurrent jobs within practical time budget constraints on commodity hardware.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑