我们在机器学习的突变测试中真正测试的是什么？批判性反思
What Are We Really Testing in Mutation Testing for Machine Learning? A Critical Reflection
摘要：
变异测试是一种成熟的技术，用于通过将人工错误注入到生产代码中来评估测试套件的质量。近年来，突变测试已经扩展到机器学习（ML）系统，特别是深度学习（DL）;研究人员已经提出了方法，工具和统计上合理的方法来确定DL系统中的突变体是否被杀死。然而，正如我们将在这项工作中讨论的那样，可以提出的问题是，目前在DL中使用的突变测试技术在多大程度上实际上符合突变测试的经典解释。我们观察到ML模型开发类似于测试驱动的开发（TDD）过程，其中训练算法（“程序员”）生成一个模型（程序），该模型将数据点（测试数据）拟合到标签（隐式断言），直到某个阈值。然而，考虑到在这种TDD隐喻下提出的ML系统的突变测试技术，在当前的方法中，生产代码和测试代码之间的区别是模糊的，并且突变操作符的现实性可能受到挑战。我们还考虑了经典突变测试的基本假设：有能力的程序员假设和耦合效应假设。正如我们将要说明的那样，这些假设并不能简单地转化为ML系统开发，需要更有意识和明确的范围和概念映射才能真正得出相似之处。根据我们的观察，我们提出了几个行动点，以便更好地将ML的突变测试技术与经典突变测试的范式和词汇表对齐。
英文摘要：
Mutation testing is a well-established technique for assessing a test suite's quality by injecting artificial faults into production code. In recent years, mutation testing has been extended to machine learning (ML) systems, and deep learning (DL) in particular; researchers have proposed approaches, tools, and statistically sound heuristics to determine whether mutants in DL systems are killed or not. However, as we will argue in this work, questions can be raised to what extent currently used mutation testing techniques in DL are actually in line with the classical interpretation of mutation testing. We observe that ML model development resembles a test-driven development (TDD) process, in which a training algorithm (`programmer') generates a model (program) that fits the data points (test data) to labels (implicit assertions), up to a certain threshold. However, considering proposed mutation testing techniques for ML systems under this TDD metaphor, in current approaches, the distinction between production and test code is blurry, and the realism of mutation operators can be challenged. We also consider the fundamental hypotheses underlying classical mutation testing: the competent programmer hypothesis and coupling effect hypothesis. As we will illustrate, these hypotheses do not trivially translate to ML system development, and more conscious and explicit scoping and concept mapping will be needed to truly draw parallels. Based on our observations, we propose several action points for better alignment of mutation testing techniques for ML with paradigms and vocabularies of classical mutation testing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
