通过终身实验数据库(LDE)实现再现性和元学习
Enabling Reproducibility and Meta-learning Through a Lifelong Database of Experiments (LDE)
摘要:
人工智能(AI)的开发本质上是迭代和实验性的。在正常的开发过程中,特别是随着自动化人工智能的出现,会产生数百或数千个实验,这些实验往往会丢失或永远不会再次检查。我们失去了记录这些实验并从中大规模学习的机会,但跟踪和复制这些实验的复杂性往往让数据科学家望而却步。我们提出了终身实验数据库(LDE),自动提取和存储链接的元数据从实验工件,并提供功能,以重现这些工件,并执行跨它们的元学习。我们存储来自AI开发生命周期多个阶段的上下文,包括数据集、管道、每个阶段的配置方式,以及训练运行时的运行环境信息。存储的元数据的标准化特性允许查询和聚合,特别是在通过性能度量对工件进行排名方面。我们展示了LDE的能力,再现现有的元学习研究和存储在我们的系统中再现的元数据。然后,我们在此元数据上执行两个实验:1)检查性能指标的可重复性和可变性,以及2)在数据之上实现一些元学习算法,并检查实验结果的可变性如何影响推荐性能。实验结果表明,性能存在显著差异,特别是取决于数据集配置;当元学习建立在结果之上时,这种差异会继续存在,当使用聚合结果时,性能会提高。这表明,自动收集和聚合结果的系统(如LDE)不仅有助于实现元学习,而且还可以提高其性能。
英文摘要:
Artificial Intelligence (AI) development is inherently iterative and experimental. Over the course of normal development, especially with the advent of automated AI, hundreds or thousands of experiments are generated and are often lost or never examined again. There is a lost opportunity to document these experiments and learn from them at scale, but the complexity of tracking and reproducing these experiments is often prohibitive to data scientists. We present the Lifelong Database of Experiments (LDE) that automatically extracts and stores linked metadata from experiment artifacts and provides features to reproduce these artifacts and perform meta-learning across them. We store context from multiple stages of the AI development lifecycle including datasets, pipelines, how each is configured, and training runs with information about their runtime environment. The standardized nature of the stored metadata allows for querying and aggregation, especially in terms of ranking artifacts by performance metrics. We exhibit the capabilities of the LDE by reproducing an existing meta-learning study and storing the reproduced metadata in our system. Then, we perform two experiments on this metadata: 1) examining the reproducibility and variability of the performance metrics and 2) implementing a number of meta-learning algorithms on top of the data and examining how variability in experimental results impacts recommendation performance. The experimental results suggest significant variation in performance, especially depending on dataset configurations; this variation carries over when meta-learning is built on top of the results, with performance improving when using aggregated results. This suggests that a system that automatically collects and aggregates results such as the LDE not only assists in implementing meta-learning but may also improve its performance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。