arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.15079cs.LGcs.DBcs.DLcs.IR
收藏

饼干师:用于可发现、可治理和可重用的机器学习数据集元数据生成

Croissant Baker: Metadata Generation for Discoverable, Governable, and Reusable ML Datasets

Technical University of Munich(慕尼黑技术大学);Massachusetts Institute of Technology(麻省理工学院);Helmholtz Munich(海德堡-慕尼黑研究所以及医学中心);Barcelona Supercomputing Center(巴塞罗那超级计算中心);Google(谷歌);Couchbase(Couchbase公司);University of Maryland, Baltimore County(马里兰大学巴尔的摩县分校);Nutanix(Nutanix公司);Georg-August-University Göttingen(哥廷根大学);Salesforce(Salesforce公司);Sage Bionetworks(Sage Bionetworks公司);Dotphoton(Dotphoton公司);Harvard University(哈佛大学);Eindhoven University of Technology(埃因霍温理工大学);Bayer AG(拜耳公司);Independent Researcher(独立研究者);Massachusetts General Hospital(麻省总医院);Columbia University(哥伦比亚大学)
Rafi Al Attrach, Rajna Fani, Sebastian Lobentanzer, Joan Giner-Miguelez, Debanshu Das, Varuni H. K., Nobin Sarwar, Rajat Ghosh, Anwai Archit, Surbhi Motghare, C… 展开作者

Rafi Al Attrach, Rajna Fani, Sebastian Lobentanzer, Joan Giner-Miguelez, Debanshu Das, Varuni H. K., Nobin Sarwar, Rajat Ghosh, Anwai Archit, Surbhi Motghare, Christina Conrad Parry, Luis Oala, Lara Grosso, Joaquin Vanschoren, Steffen Vogler, Sujata Goswami, Eric S. Rosenthal, Marzyeh Ghassemi, Matthew McDermott, Tom Pollard

AI总结:

本文提出Croissant Baker,一种本地优先的开源命令行工具,可直接从数据集目录生成验证过的Croissant元数据,通过模块化处理程序注册表,支持大规模数据集,如MIMIC-IV,达到97-100%的准确率。

摘要:

Croissant已成为机器学习数据集的元数据标准,提供一种结构化的JSON-LD格式,使数据集发现、自动摄入和可重复分析在不同ML平台间可机器验证。NeurIPS现在要求每个提交的data set tracks都包含Croissant元数据。然而,实践中生成Croissant元数据通常从上传到公共平台开始,这在受控和大型本地存储库中不可行,这些存储库持有ML日益依赖的高价值数据。我们发布了Croissant Baker,一种本地优先、开源的命令行工具,通过模块化处理程序注册表直接从数据集目录生成验证过的Croissant元数据。我们在超过140个数据集中评估了Croissant Baker,扩展到MIMIC-IV的8.86亿行和374个Parquet文件。在与生产者编写或标准衍生的地面真相的对比中,Croissant Baker在多个领域达到97-100%的一致性。

英文摘要:

Croissant has emerged as the metadata standard for machine learning datasets, providing a structured, JSON-LD-based format that makes dataset discovery, automated ingestion, and reproducible analysis machine-checkable across ML platforms. Adoption has accelerated, and NeurIPS now requires Croissant metadata in every submission to its dataset tracks. Yet in practice Croissant generation usually starts with uploading data to a public platform, a path infeasible for governed and large local repositories that hold much of the high-value data ML increasingly relies on. We release Croissant Baker, a local-first, open-source command-line tool that generates validated Croissant metadata directly from a dataset directory through a modular handler registry. We evaluate Croissant Baker on over 140 datasets, scaling to MIMIC-IV at 886 million rows and 374 Parquet files. On held-out comparisons against producer-authored or standards-derived ground truth, Croissant Baker reaches 97-100% agreement across multiple domains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑