有限样本下的机器学习模型测试:工业真空泵应用
Testing of Machine Learning Models with Limited Samples: An Industrial Vacuum Pumping Application
摘要:
在工业生产中,通常缺乏用于机器学习(ML)分类和回归模型的训练数据,特别是对于耗时或稀疏运行的制造过程。大多数有限的地面实况数据用于训练,而少数样本用于测试。在这里,测试样本的数量不足以正确评估分类和回归测试中ML模型的鲁棒性。此外,如果输入数据与预期不同,这些ML模型的输出可能不准确甚至失败。这是用于精炼钢工业中的电渣重熔(ESR)过程中的ML模型的情况,以预测真空室中的压力。每个工作日发生一次真空抽吸事件,在一年的抽吸中产生几百个样本用于训练和测试。在缺乏足够的训练和测试样本的情况下,本文首先提出了一种基于真空泵原理的新的增强样本集的生成方法。基于生成的增强样本,提出了三个测试场景和一个测试预言,以评估用于工业规模生产的ML模型的鲁棒性。利用Uddeholms AB钢铁公司的实际工业生产数据进行了实验。评估表明,Entrance和神经网络是最强大的增强数据训练时,使用建议的测试策略。评估也表明了该方法的有效性,在检查和提高ML算法的鲁棒性,在这种情况下。这项工作改进了软件测试在类似环境中的最先进的鲁棒性测试。最后,本文提出了一种MLOps实现方法,用于边缘节点上的实时ML模型预测和操作,以及从云中自动连续交付ML软件。
英文摘要:
There is often a scarcity of training data for machine learning (ML) classification and regression models in industrial production, especially for time-consuming or sparsely run manufacturing processes. A majority of the limited ground-truth data is used for training, while a handful of samples are left for testing. Here, the number of test samples is inadequate to properly evaluate the robustness of the ML models under test for classification and regression. Furthermore, the output of these ML models may be inaccurate or even fail if the input data differ from the expected. This is the case for ML models used in the Electroslag Remelting (ESR) process in the refined steel industry to predict the pressure in a vacuum chamber. A vacuum pumping event that occurs once a workday generates a few hundred samples in a year of pumping for training and testing. In the absence of adequate training and test samples, this paper first presents a method to generate a fresh set of augmented samples based on vacuum pumping principles. Based on the generated augmented samples, three test scenarios and one test oracle are presented to assess the robustness of an ML model used for production on an industrial scale. Experiments are conducted with real industrial production data obtained from Uddeholms AB steel company. The evaluations indicate that Ensemble and Neural Network are the most robust when trained on augmented data using the proposed testing strategy. The evaluation also demonstrates the proposed method's effectiveness in checking and improving ML algorithms' robustness in such situations. The work improves software testing's state-of-the-art robustness testing in similar settings. Finally, the paper presents an MLOps implementation of the proposed approach for real-time ML model prediction and action on the edge node and automated continuous delivery of ML software from the cloud.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。