首页/论文/1908.04705

arXiv 1908.047052019-08cs.LGcs.DCcs.PFstat.ML

Exploiting Parallelism Opportunities with Deep Learning Frameworks

Harvard University（哈佛大学）；Facebook

Yu Emma Wang, Carole-Jean Wu, Xiaodong Wang, Kim Hazelwood, David Brooks

英文摘要：

State-of-the-art machine learning frameworks support a wide variety of design features to enable a flexible machine learning programming interface and to ease the programmability burden on machine learning developers. Identifying and using a performance-optimal setting in feature-rich frameworks, however, involves a non-trivial amount of performance profiling efforts and often relies on domain-specific knowledge. This paper takes a deep dive into analyzing the performance impact of key design features in a machine learning framework and quantifies the role of parallelism. The observations and insights distill into a simple set of guidelines that one can use to achieve much higher training and inference speedup. Across a diverse set of real-world deep learning models, the evaluation results show that the proposed performance tuning guidelines outperform the Intel and TensorFlow recommended settings by 1.29x and 1.34x, respectively.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Para-Pipe：在片上系统（SoC）上利用机器学习计算图的分层算子并行性

Para-Pipe: Exploiting Hierarchical Operator Parallelism of ML Computational Graphs on SoCs

School of Computing, National University of Singapore（新加坡国立大学计算机学院）；the University of Amsterdam（阿姆斯特丹大学）；Black Sesame Technologies（黑芝麻智能科技）

Para-Pipe是整合算子并行性的分层映射框架，可平衡SoC上深度学习的吞吐量与延迟，提升能效，在两款SoC上分别实现11.0%和23.3%的能效提升。

2026-09-04cs.DCcs.LGcs.PF

HyPar-Flow: Exploiting MPI and Keras for Scalable Hybrid-Parallel DNN Training using TensorFlow

The Ohio State University（俄亥俄州立大学）

To reduce training time of large-scale DNNs, scientists have started to explore parallelization strategies like data-parallelism, model-parallelism, and hybrid-parallelism. While data-parallelism has been extensively studied and developed, several problems exist in realizing model-parallelism and hybrid-parallelism efficiently. Four major problems we focus on are: 1) defining a notion of a distributed model across processes, 2) implementing forward/back-propagation across process boundaries that requires explicit communication, 3) obtaining parallel speedup on an inherently sequential task, an…

2020-02-19cs.AIcs.DCcs.LG

Improving Strong-Scaling of CNN Training by Exploiting Finer-Grained Parallelism

Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）

Scaling CNN training is necessary to keep up with growing datasets and reduce training time. We also see an emerging need to handle datasets with very large samples, where memory requirements for training are large. Existing training frameworks use a data-parallel approach that partitions samples within a mini-batch, but limits to scaling the mini-batch size and memory consumption makes this untenable for large samples. We describe and implement new approaches to convolution, which parallelize using spatial decomposition or a combination of sample and spatial decomposition. This introduces man…

2019-03-15cs.DCcs.LG

利用时间并行性加速FPGA上的LSTM自编码器

Exploiting temporal parallelism for LSTM Autoencoder acceleration on FPGA

本文提出一种基于FPGA的数据流架构，通过时间并行性实现多层LSTM-AE的并发处理，实验显示在CPU和GPU上具有显著的加速效果和能效提升。

2026-03-14cs.ARcs.LG

Primitives for Dynamic Big Model Parallelism

School of Computer Science Carnegie Mellon University（卡内基梅隆大学计算机科学学院）

When training large machine learning models with many variables or parameters, a single machine is often inadequate since the model may be too large to fit in memory, while training can take a long time even with stochastic updates. A natural recourse is to turn to distributed cluster computing, in order to harness additional memory and processors. However, naive, unstructured parallelization of ML algorithms can make inefficient use of distributed memory, while failing to obtain proportional convergence speedups - or can even result in divergence. We develop a framework of primitives for dyna…

2014-06-18cs.DCcs.LGstat.ML

利用剪枝神经网络中的稀疏性优化大规模模型训练

Exploiting Sparsity in Pruned Neural Networks to Optimize Large Model Training

University of Maryland（马里兰大学）

提出利用剪枝神经网络的稀疏性优化数据并行和层间并行中的内存与通信，集成于AxoNN框架，在512块V100 GPU上实现27亿参数模型内存减少74%、通信时间减少40%，整体加速34%。

2023-05-14cs.AIcs.DCcs.LG

更多相关论文收起更多论文

利用学生并行性实现在线服务中类BERT模型的高效GPU推理

Exploiting Student Parallelism for Efficient GPU Inference of BERT-like Models in Online Services

Tencent（腾讯）；Hong Kong University of Science and Technology（香港科技大学）；Xiamen University（厦门大学）；Macau University of Science and Technology（澳门科技大学）；Zhongguancun Lab.（中关村实验室）

本文提出\sys，通过堆叠蒸馏、提升集成与自适应学生剪枝，将深层类BERT模型转化为并行浅层学生模型，在保持精度的同时显著降低在线GPU推理延迟并提升突发负载吞吐量。

2025-03-18cs.LG

PAPI：利用支持存内计算的计算系统开发大语言模型解码中的动态并行性

PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing System

Institute of Computing Technology, CAS（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；King’s College London（伦敦国王学院）；ETH Zürich（苏黎世联邦理工学院）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；NVIDIA（英伟达公司）

PAPI提出一种支持存内计算的异构架构，通过在线内核表征和动态调度，加速大语言模型解码，在三个LLM上分别比异构和纯PIM加速器快1.8倍和11.1倍。

2025-02-27cs.AIcs.ARcs.DC

利用联邦学习中的相关性：机遇与实际限制

Exploiting Correlations in Federated Learning: Opportunities and Practical Limitations

Department of Electrical Engineering (ISY), Linköping University（_linköping大学电气工程系）；CentraleSupélec, CNRS, L2S, Université Paris-Saclay（_centralesupelec, CNRS, L2S, 巴黎萨克雷大学）；Division of Decision and Control Systems, School of EECS, KTH Royal Institute of Technology（_决策与控制系统系，皇家理工学院）

本文系统分类了梯度和模型压缩方案，探讨了任务复杂度、模型架构对相关性的影响，并提出自适应压缩方法以提升联邦学习效率。

2026-04-16cs.DCcs.ITcs.LG

Survey: Exploiting Data Redundancy for Optimization of Deep Learning

North Carolina State University（北卡罗来纳州立大学）；William & Mary（威廉与玛丽学院）；Northeastern University（东北大学）

Data redundancy is ubiquitous in the inputs and intermediate results of Deep Neural Networks (DNN). It offers many significant opportunities for improving DNN performance and efficiency and has been explored in a large body of work. These studies have scattered in many venues across several years. The targets they focus on range from images to videos and texts, and the techniques they use to detect and exploit data redundancy also vary in many aspects. There is not yet a systematic examination and summary of the many efforts, making it difficult for researchers to get a comprehensive view of t…

2022-08-29cs.LG

FinXplore：一种用于平衡与发现投资机会的自适应深度强化学习框架

FinXplore: An Adaptive Deep Reinforcement Learning Framework for Balancing and Discovering Investment Opportunities

School of Mathematical & \& Statistical Sciences（数学与统计科学学院）；Indian Institute of Technology Mandi（印度理工学院曼迪分校）

本研究提出FinXplore框架，通过两个DRL代理动态平衡现有资产利用与新投资机会探索，显著提升投资组合优化表现。

2025-09-05cs.AIcs.LG

Towards Deep Learning Models for Psychological State Prediction using Smartphone Data: Challenges and Opportunities

University of Oxford（牛津大学）；The Alan Turing Institute（艾伦·图灵研究所）；University of Warwick（华威大学）；University College London（伦敦大学学院）

There is an increasing interest in exploiting mobile sensing technologies and machine learning techniques for mental health monitoring and intervention. Researchers have effectively used contextual information, such as mobility, communication and mobile phone usage patterns for quantifying individuals' mood and wellbeing. In this paper, we investigate the effectiveness of neural network models for predicting users' level of stress by using the location information collected by smartphones. We characterize the mobility patterns of individuals using the GPS metrics presented in the literature an…

2017-11-16cs.AIcs.LGstat.ML

Deep Learning Recommendation Model for Personalization and Recommendation Systems

Northwestern University（西北大学）；Harvard University（哈佛大学）；Facebook（脸书）

With the advent of deep learning, neural network-based recommendation models have emerged as an important tool for tackling personalization and recommendation tasks. These networks differ significantly from other deep learning networks due to their need to handle categorical features and are not well studied or understood. In this paper, we develop a state-of-the-art deep learning recommendation model (DLRM) and provide its implementation in both PyTorch and Caffe2 frameworks. In addition, we design a specialized parallelization scheme utilizing model parallelism on the embedding tables to mit…

2019-05-31cs.IRcs.LG