arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.08799cs.HCcs.LG
收藏

Wizard of Errors:介绍和评估机器学习错误在绿野仙踪研究

Wizard of Errors: Introducing and Evaluating Machine Learning Errors in Wizard of Oz Studies

Eindhoven University of Technology(埃因霍温理工大学)

Anniek Jansen, Sara Colombo

摘要:

在设计支持机器学习(ML)的解决方案时,设计人员通常需要通过绿野仙踪(WoZ)方法模拟ML行为,以便在ML模型可用之前测试用户体验。虽然再现ML错误对于具有良好的表示是必不可少的,但它们很少被考虑。我们介绍错误向导(WoE),这是一个用于对支持ML的解决方案进行WoZ研究的工具,允许在用户体验评估期间模拟ML错误。我们探讨了如何使用该系统来模拟计算机视觉模型的行为。我们用设计专业的学生测试了WoE,以确定在设计中考虑ML错误的重要性,使用描述性错误类型而不是混淆矩阵的相关性,以及WoZ研究中手动错误控制的适用性。我们的工作确定了几个挑战,防止现实的错误表示设计师在这样的研究。我们讨论了这些研究结果的影响设计。

英文摘要:

When designing Machine Learning (ML) enabled solutions, designers often need to simulate ML behavior through the Wizard of Oz (WoZ) approach to test the user experience before the ML model is available. Although reproducing ML errors is essential for having a good representation, they are rarely considered. We introduce Wizard of Errors (WoE), a tool for conducting WoZ studies on ML-enabled solutions that allows simulating ML errors during user experience assessment. We explored how this system can be used to simulate the behavior of a computer vision model. We tested WoE with design students to determine the importance of considering ML errors in design, the relevance of using descriptive error types instead of confusion matrix, and the suitability of manual error control in WoZ studies. Our work identifies several challenges, which prevent realistic error representation by designers in such studies. We discuss the implications of these findings for design.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑