挑战AI思维：一个用于主动AI测试的群体系统
Challenge AI Mind: A Crowd System for Proactive AI Testing
摘要：
人工智能（AI）已经在各个方面深入到我们的生活中;然而，如果没有适当的测试，部署的AI系统经常被批评在关键和尴尬的情况下失败。现有的测试方法主要依赖于固定的和预定义的数据集，提供有限的测试覆盖范围。在本文中，我们提出了主动测试的概念，以动态生成测试数据和评估人工智能系统的性能。我们进一步介绍了Challenge.AI，这是一个新的人群系统，其特点是在错误生成，错误验证，错误分类和错误分析的过程中集成了众包和机器学习技术。我们与AI开发人员一起展示参与式设计的经验和见解。评价结果表明，在机器学习技术的帮助下，人群工作流更加有效。人工智能开发人员发现，我们的系统可以帮助他们发现人工智能模型产生的未知错误，并参与主动测试过程。
英文摘要：
Artificial Intelligence (AI) has burrowed into our lives in various aspects; however, without appropriate testing, deployed AI systems are often being criticized to fail in critical and embarrassing cases. Existing testing approaches mainly depend on fixed and pre-defined datasets, providing a limited testing coverage. In this paper, we propose the concept of proactive testing to dynamically generate testing data and evaluate the performance of AI systems. We further introduce Challenge.AI, a new crowd system that features the integration of crowdsourcing and machine learning techniques in the process of error generation, error validation, error categorization, and error analysis. We present experiences and insights into a participatory design with AI developers. The evaluation shows that the crowd workflow is more effective with the help of machine learning techniques. AI developers found that our system can help them discover unknown errors made by the AI models, and engage in the process of proactive testing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
