arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2009.00449cs.HC
收藏

用AutoML系统评价探索性模型建立过程

Towards Evaluating Exploratory Model Building Process with AutoML Systems

Sungsoo Ray Hong, Sonia Castelo, Vito D'Orazio, Christopher Benthune, Aecio Santos, Scott Langevin, David Jonker, Enrico Bertini, Juliana Freire

摘要:

自动机器学习(AutoML)系统的使用是高度开放和探索性的。虽然严格评估最终用户如何与AutoML交互至关重要,但为这种探索性系统建立一个强大的评估方法是具有挑战性的。首先,AutoML很复杂,包括多个子组件,这些子组件支持用于合成ML管道的各种子任务,例如数据准备,问题规范和模型生成,因此很难产生告诉我们哪些组件成功或失败的见解。其次,由于AutoML的使用模式是高度探索性的,因此不可能仅仅依赖于广泛使用的任务效率和有效性指标作为成功指标。为了解决评估中的挑战,我们提出了一种评估方法,该方法(1)指导AutoML构建者将其AutoML系统划分为多个子系统组件,(2)通过可视化最终用户的行为模式和态度数据帮助他们对每个组件进行推理。我们进行了一项研究,以了解何时,如何,为什么以及应用我们的方法可以帮助构建者更好地了解他们的系统和最终用户。我们招募了3个专业的AutoML构建者团队。这些小组准备了自己的系统,并让41个最终用户使用这些系统。使用我们的方法,我们可视化了最终用户的行为和态度数据,并将结果分发给团队。我们从两个方面分析了结果:AutoML构建者从最终用户那里学到了什么类型的新见解,以及(2)评估方法如何帮助构建者理解工作流程及其系统的有效性。我们的研究结果提出了新的见解,解释了AutoML领域未来的设计机会,以及如何使用我们的方法帮助构建者确定见解,并让他们绘制改进系统的具体方向。

英文摘要:

The use of Automated Machine Learning (AutoML) systems are highly open-ended and exploratory. While rigorously evaluating how end-users interact with AutoML is crucial, establishing a robust evaluation methodology for such exploratory systems is challenging. First, AutoML is complex, including multiple sub-components that support a variety of sub-tasks for synthesizing ML pipelines, such as data preparation, problem specification, and model generation, making it difficult to yield insights that tell us which components were successful or not. Second, because the usage pattern of AutoML is highly exploratory, it is not possible to rely solely on widely used task efficiency and effectiveness metrics as success metrics. To tackle the challenges in evaluation, we propose an evaluation methodology that (1) guides AutoML builders to divide their AutoML system into multiple sub-system components, and (2) helps them reason about each component through visualization of end-users' behavioral patterns and attitudinal data. We conducted a study to understand when, how, why, and applying our methodology can help builders to better understand their systems and end-users. We recruited 3 teams of professional AutoML builders. The teams prepared their own systems and let 41 end-users use the systems. Using our methodology, we visualized end-users' behavioral and attitudinal data and distributed the results to the teams. We analyzed the results in two directions: what types of novel insights the AutoML builders learned from end-users, and (2) how the evaluation methodology helped the builders to understand workflows and the effectiveness of their systems. Our findings suggest new insights explaining future design opportunities in the AutoML domain as well as how using our methodology helped the builders to determine insights and let them draw concrete directions for improving their systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑