ML-EXray：了解边缘ML部署
ML-EXray: Visibility into ML Deployment on the Edge
摘要：
得益于不断扩大的云基础设施，如今的深度神经网络（DNN）在云端训练时具有越来越高的性能。研究人员花了几个月的时间来争取额外的几个百分点的模型准确性。然而，当这些模型实际部署在边缘设备上时，通常情况下，性能会突然下降10%以上，而没有明显的原因。关键的挑战是，对边缘设备上的ML推理执行没有太多的可见性，并且对边缘部署过程中的潜在问题知之甚少。我们提出了ML-EXray，这是一个端到端框架，它提供了ML执行的层级细节的可见性，并帮助开发人员分析和调试云到边缘部署问题。通常情况下，边缘性能次优的原因不仅在于模型本身，还在于整个数据流和部署过程中的每个操作。评估表明，ML-EXray可以有效地捕获部署问题，例如预处理错误，量化问题，次优内核等，使用ML-EXray，用户只需编写不到15行代码即可全面检查边缘部署管道。ML-EXray可以消除这些问题，将模型性能校正高达30%，精确定位易出错的层，并指导用户将内核执行延迟优化两个数量级。代码和API将作为开源多语言工具库和Python部署验证库发布。
英文摘要：
Benefiting from expanding cloud infrastructure, deep neural networks (DNNs) today have increasingly high performance when trained in the cloud. Researchers spend months of effort competing for an extra few percentage points of model accuracy. However, when these models are actually deployed on edge devices in practice, very often, the performance can abruptly drop over 10% without obvious reasons. The key challenge is that there is not much visibility into ML inference execution on edge devices, and very little awareness of potential issues during the edge deployment process. We present ML-EXray, an end-to-end framework, which provides visibility into layer-level details of the ML execution, and helps developers analyze and debug cloud-to-edge deployment issues. More often than not, the reason for sub-optimal edge performance does not only lie in the model itself, but every operation throughout the data flow and the deployment process. Evaluations show that ML-EXray can effectively catch deployment issues, such as pre-processing bugs, quantization issues, suboptimal kernels, etc. Using ML-EXray, users need to write less than 15 lines of code to fully examine the edge deployment pipeline. Eradicating these issues, ML-EXray can correct model performance by up to 30%, pinpoint error-prone layers, and guide users to optimize kernel execution latency by two orders of magnitude. Code and APIs will be released as an open-source multi-lingual instrumentation library and a Python deployment validation library.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
