人类智能与机器故障:集成机器学习系统故障排除
On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems
摘要:
我们研究了依赖于不同组件的分析管道的机器学习系统的故障排除问题。理解和修复这种集成系统中出现的错误是困难的,因为故障可能发生在执行工作流程中的多个点。此外,错误可能会传播、扩大或被抑制,使责任分配变得困难。我们提出了一个人在回路中的方法,利用人类的智慧来排除系统故障。该方法通过人工计算任务模拟潜在的组件修复,并测量系统整体行为的预期改进。该方法为设计人员提供了如何最好地改进系统的指导。我们证明了自动图像描述系统,已按下到现实世界中使用的方法的有效性。
英文摘要:
We study the problem of troubleshooting machine learning systems that rely on analytical pipelines of distinct components. Understanding and fixing errors that arise in such integrative systems is difficult as failures can occur at multiple points in the execution workflow. Moreover, errors can propagate, become amplified or be suppressed, making blame assignment difficult. We propose a human-in-the-loop methodology which leverages human intellect for troubleshooting system failures. The approach simulates potential component fixes through human computation tasks and measures the expected improvements in the holistic behavior of the system. The method provides guidance to designers about how they can best improve the system. We demonstrate the effectiveness of the approach on an automated image captioning system that has been pressed into real-world use.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。