DNN运行准确率的迭代评估与改进
Iterative Assessment and Improvement of DNN Operational Accuracy
AI总结:
提出DAIC方法,结合低成本在线伪预言机与高成本离线采样技术,在DNN生命周期迭代中评估并改进运行准确率。
摘要:
深度神经网络(DNN)凭借其在特定任务中类人甚至超人的性能,如今在许多应用领域被广泛采用。然而,由于不可预测或未考虑的运行条件,现场会出现意外故障,使得DNN在运行中的性能与发布前估计的性能大相径庭。在DNN系统的生命周期中,准确率评估通常通过两种方式解决:离线方式,通过对运行输入进行采样;或在线方式,通过伪预言机。前者由于需要对采样的输入进行手动标记,被认为成本更高。后者是自动的但准确度较低。我们认为,像MLOps这样新兴的适用于机器学习系统的迭代工业级生命周期模型,提供了利用运行中观察到的输入的可能性,不仅能提供DNN准确率的可靠估计,还能通过重构/重训练操作来改进它。我们提出了DAIC(DNN评估与改进周期),这是一种结合“低成本”在线伪预言机和“高成本”离线采样技术的方法,以在其生命周期的迭代中估计和改进DNN的运行准确率。初步结果显示了结合这两种方法并将其集成到DNN生命周期中的益处。
英文摘要:
Deep Neural Networks (DNN) are nowadays largely adopted in many application domains thanks to their human-like, or even superhuman, performance in specific tasks. However, due to unpredictable/unconsidered operating conditions, unexpected failures show up on field, making the performance of a DNN in operation very different from the one estimated prior to release. In the life cycle of DNN systems, the assessment of accuracy is typically addressed in two ways: offline, via sampling of operational inputs, or online, via pseudo-oracles. The former is considered more expensive due to the need for manual labeling of the sampled inputs. The latter is automatic but less accurate. We believe that emerging iterative industrial-strength life cycle models for Machine Learning systems, like MLOps, offer the possibility to leverage inputs observed in operation not only to provide faithful estimates of a DNN accuracy, but also to improve it through remodeling/retraining actions. We propose DAIC (DNN Assessment and Improvement Cycle), an approach which combines ''low-cost'' online pseudo-oracles and ''high-cost'' offline sampling techniques to estimate and improve the operational accuracy of a DNN in the iterations of its life cycle. Preliminary results show the benefits of combining the two approaches and integrating them in the DNN life cycle.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。