scida:面向科学大数据的可扩展分析
scida: scalable analysis for scientific big data
AI总结:
本文介绍 Python 包 scida,它借助 dask 任务图支持大型宇宙学与星系形成模拟数据的并行、核外和按需可扩展分析。
摘要:
scida 是一个用于读取和分析大型科学数据集的 Python 软件包,开箱即用地支持多种宇宙学和星系形成模拟。只需调用简单的 load() 函数,即可通过层级化、类似字典的数据结构访问数据。该软件包利用 dask 库实现可扩展、并行和核外计算:用户会话中的所有计算请求会先被收集到任务图中,既可执行任意自定义分析,也可执行所有可用的 dask(数组)操作。后续计算仅在收到请求时,才在目标资源(例如 HPC 集群)上执行。
英文摘要:
scida is a Python package for reading and analyzing large scientific data sets with support for various cosmological and galaxy formation simulations out-of-the-box. Data access is provided through a hierarchical dictionary-like data structure after a simple load() function. Using the dask library for scalable, parallel and out-of-core computation, all computation requests from a user session are first collected in a task graph. Arbitrary custom analysis, as well as all available dask (array) operations, can be performed. The subsequent computation is executed only upon request, on a target resource (e.g. a HPC cluster).
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。