arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.02771cs.AIcs.CE
收藏

科学人工智能的自动化数据准备

Automated Data Readiness for Scientific AI

ORNL(橡树岭国家实验室)
Sean R. Wilkinson, Valentine G. Anantharaj, Jong Youl Choi, Ketan Maheshwari, Marshall McDonnell, Massimiliano Lupo Pasini, Polina Shpilker, Renan Souza, Patric… 展开作者

Sean R. Wilkinson, Valentine G. Anantharaj, Jong Youl Choi, Ketan Maheshwari, Marshall McDonnell, Massimiliano Lupo Pasini, Polina Shpilker, Renan Souza, Patrick Widener, Sarp Oral, Wesley Brewer

AI总结:

研究针对科学数据集转化为AI训练数据的问题,提出REDI开源框架,经五阶段管道处理,配套工具自动化合规性与编目发布,贡献是跨域提供自动化数据准备,转化数据瓶颈。

摘要:

领导计算设施管理大规模科学数据集,现有框架未完全统一自动化转换等。我们提出REDI开源框架,通过统一五阶段管道解决这一差距,各阶段有工具用于可重复性和部署。经多领域评估,REDI将数据集转化为AI可用,结果表明其是跨域平台,可转化数据瓶颈。

英文摘要:

Leadership computing facilities steward large-scale scientific datasets that routinely require substantial transformation before serving as AI training data. However, no existing framework fully unifies automated transformation, readiness assessment, provenance tracking, and agent-native deployment. We present REDI, an open-source framework that addresses this gap through a unified five-stage pipeline (ingest, preprocess, transform, structure, and output) with per-stage instrumentation for reproducibility and deployment as an agent-callable skill; companion tool SetGo automates FAIR compliance and catalog publication. Evaluated across climate, proteomics, materials science, and nuclear fusion, REDI transforms all datasets from raw to AI-ready, with outputs validated against domain-expert references, and preliminary results show near-ideal parallel scaling to 100 nodes on Frontier for the climate case. Provenance-instrumented profiling reveals file I/O as the dominant pipeline cost, with format selection a first-order optimization lever. These results establish REDI as a cross-domain platform providing automated data readiness for scientific AI, transforming data preparation bottlenecks into reproducible, reusable community assets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑