Deploying deep learning (DL) on mobile devices has been a notable trend in recent years. To support fast inference of on-device DL, DL libraries play a critical role as algorithms and hardware do. Unfortunately, no prior work ever dives deep into the ecosystem of modern DL libs and provides quantitative results on their performance. In this paper, we first build a comprehensive benchmark that includes 6 representative DL libs and 15 diversified DL models. We then perform extensive experiments on 10 mobile devices, which help reveal a complete landscape of the current mobile DL libs ecosystem. For example, we find that the best-performing DL lib is severely fragmented across different models and hardware, and the gap between those DL libs can be rather huge. In fact, the impacts of DL libs can overwhelm the optimizations from algorithms or hardware, e.g., model quantization and GPU/DSP-based heterogeneous computing. Finally, atop the observations, we summarize practical implications to different roles in the DL lib ecosystem.
MobiFlow: Real-World Mobile Agent Benchmarking through Trajectory Fusion
Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University（上海交通大学并行与分布式系统研究所（IPADS））；Department of Precision Instrument, Tsinghua University（清华大学精密仪器系）；National Innovation Institute of High-end Smart Appliances（国家高端智能家电创新研究院）
HealthSLM-Bench: Benchmarking Small Language Models for Mobile and Wearable Healthcare Monitoring
School of Computing and Information Systems, University of Melbourne, Australia（墨尔本大学计算机与信息系统学院）；School of Computer Science, University of Auckland, New Zealand（奥克兰大学计算机科学学院）
DIG: A Turnkey Library for Diving into Graph Deep Learning Research
Texas A&M University（得克萨斯农工大学）；Department of Computer Science and Engineering（计算机科学与工程系）
虽然存在几个用于图上的深度学习的库，但它们旨在实现图深度学习的基本操作。在研究社区中，使用现有库实现和基准测试各种高级任务仍然是痛苦和耗时的。为了促进图深度学习研究，我们引入了DIG：Dive into Graphs，这是一个交钥匙库，为更高级别的面向研究的图深度学习任务提供了统一的测试平台。目前，我们考虑图生成，图上的自监督学习，图神经网络的可解释性，以及3D图上的深度学习。对于每个方向，我们提供统一的数据接口，通用算法和评估指标的实现。总而言之，DIG是一个可扩展的，开源的和交钥匙的库，供研究人员开发新方法，并使用广泛使用的数据集和评估指标轻松地与常见基线进行比较。源代码可在https://github.com/divelab/DIG上获得。
TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；AWS（亚马逊云科技）；Shanghai Jiao Tong University（上海交通大学）；UC Davis（加州大学戴维斯分校）；Cornell（康奈尔大学）
Time to Embrace Natural Language Processing (NLP)-based Digital Pathology: Benchmarking NLP- and Convolutional Neural Network-based Deep Learning Pipelines
University of Science and Technology of China（中国科学技术大学）；UNSW Sydney（新南威尔士大学）；Genmab Inc.（Genmab公司）
