arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.16274cs.DBhep-ex
收藏

HSF条件数据库参考实现

The HSF Conditions Database Reference Implementation

Ruslan Mashinistov, Lino Gerlach, Paul Laycock, Andrea Formica, Giacomo Govi, Chris Pinkenburg

AI总结:

本文介绍HSF条件数据库的服务与客户端参考实现,采用Django+PostgreSQL+POSIX文件系统的分离设计,支持高访问速率与分布式计算,已在sPHENIX实验超2.5万核环境中验证部署与扩展能力。

摘要:

条件数据是非事件数据的子集,是处理事件数据所必需的。它带来了一系列独特的挑战,即异构结构以及分布式计算带来的高访问速率。HSF条件数据库活动是一个跨实验讨论论坛,旨在吸引尽可能广泛的参与。它源自HSF社区白皮书工作,该工作旨在研究条件数据访问,来自ATLAS、Belle II和CMS的专家就此形成了通用语言,并提出了代表最佳实践的架构。在与包括核物理(NP)以及高能物理(HEP)实验在内的更广泛社区讨论后,研究人员定义了一组核心用例、功能和行为,旨在描述核心条件数据库API。本文将介绍条件数据库服务和客户端的参考实现,二者共同封装了HSF最佳实践的条件数据处理方式。服务实现选用Django,除一个方法外,所有操作均使用对象关系映射(ORM)而非直接使用SQL。用于组织条件数据的简单关系数据库架构在PostgreSQL中实现。存储条件数据有效载荷本身的任务被外包给任何符合POSIX标准的文件系统,从而实现透明迁移和冗余。至关重要的是,这种设计清晰区分了两项任务:检索描述数据处理作业所需条件数据的元数据,以及从存储中检索实际有效载荷。本文还将介绍在OKD上使用Helm的服务部署方式,以及sPHENIX实验在布鲁克海文国家实验室(BNL)运行超过2.5万个核心的扩展测试和运维经验。

英文摘要:

Conditions data is the subset of non-event data that is necessary to process event data. It poses a unique set of challenges, namely a heterogeneous structure and high access rates by distributed computing. The HSF Conditions Databases activity is a forum for cross-experiment discussions inviting as broad a participation as possible. It grew out of the HSF Community White Paper work to study conditions data access, where experts from ATLAS, Belle II, and CMS converged on a common language and proposed a schema that represents best practice. Following discussions with a broader community, including NP as well as HEP experiments, a core set of use cases, functionality and behaviour was defined with the aim to describe a core conditions database API. This paper will describe the reference implementation of both the conditions database service and the client which together encapsulate HSF best practice conditions data handling. Django was chosen for the service implementation, which uses an ORM instead of the direct use of SQL for all but one method. The simple relational database schema to organise conditions data is implemented in PostgreSQL. The task of storing conditions data payloads themselves is outsourced to any POSIX-compliant filesystem, allowing for transparent relocation and redundancy. Crucially this design provides a clear separation between retrieving the metadata describing which conditions data are needed for a data processing job, and retrieving the actual payloads from storage. The service deployment using Helm on OKD will be described together with scaling tests and operations experience from the sPHENIX experiment running more than 25k cores at BNL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑