Deep Learning (DL) has recently achieved tremendous success. A variety of DL frameworks and platforms play a key role to catalyze such progress. However, the differences in architecture designs and implementations of existing frameworks and platforms bring new challenges for DL software development and deployment. Till now, there is no study on how various mainstream frameworks and platforms influence both DL software development and deployment in practice. To fill this gap, we take the first step towards understanding how the most widely-used DL frameworks and platforms support the DL software development and deployment. We conduct a systematic study on these frameworks and platforms by using two types of DNN architectures and three popular datasets. (1) For development process, we investigate the prediction accuracy under the same runtime training configuration or same model weights/biases. We also study the adversarial robustness of trained models by leveraging the existing adversarial attack techniques. The experimental results show that the computing differences across frameworks could result in an obvious prediction accuracy decline, which should draw the attention of DL developers. (2) For deployment process, we investigate the prediction accuracy and performance (refers to time cost and memory consumption) when the trained models are migrated/quantized from PC to real mobile devices and web browsers. The DL platform study unveils that the migration and quantization still suffer from compatibility and reliability issues. Meanwhile, we find several DL software bugs by using the results as a benchmark. We further validate the results through bug confirmation from stakeholders and industrial positive feedback to highlight the implications of our study. Through our study, we summarize practical guidelines, identify challenges and pinpoint new research directions.
Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment
Flower Labs;University of Cambridge(剑桥大学);University of Auckland(奥克兰大学);University of Melbourne(墨尔本大学);Vector Institute;Fraunhofer IMS(弗劳恩霍夫应用固体物理与材料力学研究所);NetCompany;Gachon University(嘉泉大学);Owkin;Sony AI(索尼人工智能)
MatchNAS: Optimizing Edge AI in Sparse-Label Data Contexts via Automating Deep Neural Network Porting for Mobile Deployment
School of Computer Science and Engineering, University of New South Wales(新南威尔士大学计算机科学与工程学院);Faculty of Engineering & Information Technology, University of Technology Sydney(悉尼科技大学工程与信息技术学院);CSIRO’s Data61(澳大利亚联邦科学与工业研究组织Data61);University of New South Wales(新南威尔士大学)
Examining Deployment and Refinement of the VIOLA-AI Intracranial Hemorrhage Model Using an Interactive NeoMedSys Platform
Computational Radiology & Artificial Intelligence Unit, Department of Physics and Computational Radiology, Clinic for Radiology and Nuclear Medicine(计算放射学与人工智能单元,物理与计算放射学部门,放射科与核医学诊所);Oslo University Hospital(奥斯陆大学医院);Department of Physics(物理部门);University of Oslo(奥斯陆大学);Department of Medical Biophysics(医学生物物理学部门);University of Toronto(多伦多大学)
An Orchestrated Empirical Study on Deep Learning Frameworks and Platforms
Tianjin University(天津大学);Nanyang Technological University(南洋理工大学);Harbin Institute of Technology(哈尔滨工业大学);Kyushu University(九州大学);Monash University(莫纳什大学)