To make machine learning (ML) sustainable and apt to run on the diverse devices where relevant data is, it is essential to compress ML models as needed, while still meeting the required learning quality and time performance. However, how much and when an ML model should be compressed, and {\em where} its training should be executed, are hard decisions to make, as they depend on the model itself, the resources of the available nodes, and the data such nodes own. Existing studies focus on each of those aspects individually, however, they do not account for how such decisions can be made jointly and adapted to one another. In this work, we model the network system focusing on the training of DNNs, formalize the above multi-dimensional problem, and, given its NP-hardness, formulate an approximate dynamic programming problem that we solve through the PACT algorithmic framework. Importantly, PACT leverages a time-expanded graph representing the learning process, and a data-driven and theoretical approach for the prediction of the loss evolution to be expected as a consequence of training decisions. We prove that PACT's solutions can get as close to the optimum as desired, at the cost of an increased time complexity, and that, in any case, such complexity is polynomial. Numerical results also show that, even under the most disadvantageous settings, PACT outperforms state-of-the-art alternatives and closely matches the optimal energy cost.
CANS: Accelerating Multiuser Collaborative Edge Inference via Cooperative Autodidactic NeuroSurgeon
Harbin Institute of Technology Shenzhen(哈尔滨工业大学(深圳));Politecnico di Milano(米兰理工大学);Harbin Institute of Technology(哈尔滨工业大学);Fudan University(复旦大学);Shanghai Academy of Artificial Intelligence for Science(上海人工智能科学研究院)
DVFS-Aware DNN Inference on GPUs: Latency Modeling and Performance Analysis
Beijing National Research Center for Information Science and Technology(北京信息科学国家研究中心);Department of Electronic Engineering, Tsinghua University(清华大学电子工程系)
Unlocking the potential of two-point cells for energy-efficient and resilient training of deep nets
University of Oxford(牛津大学);University of Wolverhampton(伍尔弗汉普顿大学);Edinburgh Napier University(爱丁堡龙比亚大学);University of Edinburgh(爱丁堡大学);University of Stirling(斯特灵大学)
Towards Scalable and Privacy-Preserving Deep Neural Network via Algorithmic-Cryptographic Co-design
Zhejiang University(浙江大学);Ant Group(蚂蚁集团);Macquarie University(麦考瑞大学);JZTData Technology(金智塔科技);Peking University(北京大学);East China Normal University(华东师范大学)
Asteroid: Resource-Efficient Hybrid Pipeline Parallelism for Collaborative DNN Training on Heterogeneous Edge Devices
School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);Data Science and Analytics Thrust, HKUST (GZ)(香港科技大学(广州)数据科学与分析学域);The Chinese University of Hong Kong(香港中文大学)
Bounding generalization error with input compression: An empirical study with infinite-width networks
University of Guelph(圭尔夫大学);Vector Institute for Artificial Intelligence(向量人工智能研究所);The NSF AI Institute for Artificial Intelligence and Fundamental Interactions(美国国家科学基金会人工智能与基础交互研究所);Massachusetts Institute of Technology(麻省理工学院);University of Calgary(卡尔加里大学)