arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1909.06727cs.LG
收藏

跨不同框架和平台表征深度学习开发和部署的实证研究

An Empirical Study towards Characterizing Deep Learning Development and Deployment across Different Frameworks and Platforms

Tianjin University(天津大学);Nanyang Technological University(南洋理工大学);Kyushu University(九州大学)

Qianyu Guo, Sen Chen, Xiaofei Xie, Lei Ma, Qiang Hu, Hongtao Liu, Yang Liu, Jianjun Zhao, Xiaohong Li

摘要:

深度学习(DL)最近取得了巨大的成功。各种深度学习框架和平台在促进这一进展方面发挥着关键作用。然而,现有框架和平台在架构设计和实现上的差异给DL软件开发和部署带来了新的挑战。到目前为止,还没有研究各种主流框架和平台如何在实践中影响DL软件开发和部署。为了填补这一空白,我们迈出了理解最广泛使用的DL框架和平台如何支持DL软件开发和部署的第一步。我们通过使用两种类型的DNN架构和三种流行的数据集对这些框架和平台进行了系统的研究。(1)对于开发过程,我们研究了在相同的运行时训练配置或相同的模型权重/偏差下的预测精度。我们还利用现有的对抗攻击技术研究了训练模型的对抗鲁棒性。实验结果表明,框架间的计算差异会导致预测精度明显下降,应引起DL开发人员的注意。(2)对于部署过程,我们研究了当训练的模型从PC迁移/量化到真实的移动设备和Web浏览器时的预测精度和性能(指时间成本和内存消耗)。DL平台研究揭示了迁移和量化仍然存在兼容性和可靠性问题。同时,我们发现了几个DL软件的错误,使用的结果作为基准。我们通过利益相关者的错误确认和行业的积极反馈进一步验证了结果,以突出我们研究的意义。通过我们的研究,我们总结了实践指南,确定了挑战,并指出了新的研究方向。

英文摘要:

Deep Learning (DL) has recently achieved tremendous success. A variety of DL frameworks and platforms play a key role to catalyze such progress. However, the differences in architecture designs and implementations of existing frameworks and platforms bring new challenges for DL software development and deployment. Till now, there is no study on how various mainstream frameworks and platforms influence both DL software development and deployment in practice. To fill this gap, we take the first step towards understanding how the most widely-used DL frameworks and platforms support the DL software development and deployment. We conduct a systematic study on these frameworks and platforms by using two types of DNN architectures and three popular datasets. (1) For development process, we investigate the prediction accuracy under the same runtime training configuration or same model weights/biases. We also study the adversarial robustness of trained models by leveraging the existing adversarial attack techniques. The experimental results show that the computing differences across frameworks could result in an obvious prediction accuracy decline, which should draw the attention of DL developers. (2) For deployment process, we investigate the prediction accuracy and performance (refers to time cost and memory consumption) when the trained models are migrated/quantized from PC to real mobile devices and web browsers. The DL platform study unveils that the migration and quantization still suffer from compatibility and reliability issues. Meanwhile, we find several DL software bugs by using the results as a benchmark. We further validate the results through bug confirmation from stakeholders and industrial positive feedback to highlight the implications of our study. Through our study, we summarize practical guidelines, identify challenges and pinpoint new research directions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑