Graphs and their traversal is becoming significant as it is applicable to various areas of mathematics, science and technology. Various problems in fields as varied as biochemistry (genomics), electrical engineering (communication networks), computer science (algorithms and computation) can be modeled as Graph problems. Real world scenarios including communities their interconnections and related properties can be studied using graphs. So fast, scalable, low-cost execution of parallel graph algorithms is very important. In this implementation of parallel breadth first search of graphs, we implemented Parallel BFS algorithm with 1-D partitioning of graph as described in [2] and have reduced execution time by optimizing communication for local buffers.
Breathe before Speaking: Efficient Information Dissemination Despite Noisy, Limited and Anonymous Communication
分布式计算模型通常假定处理器之间的通信可靠。虽然这样的假设通常适用于工程网络，例如，由于潜在的纠错协议，它们与生物系统的相关性非常有限，在生物系统中，消息在到达目的地之前经常被扭曲。在这项研究中，我们采取了第一步，通过严格分析由简单的代理人组成的大型匿名人群中的通信模型，通过简短和高度不可靠的消息进行交互，以减少这种差距。我们重点讨论广播问题和多数人一致性问题。两者都是分布式计算中的基本信息传播问题，其中代理的目标是收敛到某些指定的期望意见。我们开始研究这些问题中存在的通信噪声。我们的通信模型非常弱，并且遵循推送八卦通信范式：在每一轮中，希望发送信息的每个代理都会向随机的匿名代理发送消息。这种通信被进一步限制为仅包含一个比特（基本上表示意见）。最后，假设系统噪声很大，对于一些小的$ε>0$，发送的每个消息中的比特以概率$1/2-ε$独立翻转。即使在这个严格限制，随机和嘈杂的设置，我们给自然的协议，有效地解决了嘈杂的广播和嘈杂的多数共识问题。我们的协议在$O(\log n / ε^2)$轮中运行，总共使用$O(n \log n / ε^2)$消息/位，其中$n$是代理的数量。这些界限是渐近最优的，事实上，是快速和有效的消息，如果每个代理将同时直接通知代理知道规定的期望意见。
Loop scheduling techniques aim to achieve load-balanced executions of scientific applications. Dynamic loop self-scheduling (DLS) libraries for distributed-memory systems are typically MPI-based and employ a centralized chunk calculation approach (CCA) to assign variably-sized chunks of loop iterations. We present a distributed chunk calculation approach (DCA) that supports various types of DLS techniques. Using both CCA and DCA, twelve DLS techniques are implemented and evaluated in different CPU slowdown scenarios. The results show that the DLS techniques implemented using DCA outperform the…
Distributed deep learning is becoming increasingly popular due to the expanding demand for computing resources for deep learning models with a larger amount of parameters. Different from traditional training approaches, data-parallel training allows multiple compute nodes to train large deep learning models simultaneously in order to boost the training efficiency. In this paper, we present and compare six strategies for data-parallel training using PyTorch on the language model GPT-2 with 100M parameters using a qualitative approach. These strategies are Single GPU, Single Parameter Server, Di…
We present four high performance hybrid sorting methods developed for various parallel platforms: shared memory multiprocessors, distributed multiprocessors, and clusters taking advantage of existence of both shared and distributed memory. Merge sort, known for its stability, is used to design several of our algorithms. We improve its parallel performance by combining it with Quicksort. We present two models designed for shared memory MIMD (OpenMP): (a) a non-recursive Merge sort and (b) a hybrid Quicksort and Merge sort. The third model presented is designed for distributed memory MIMD (MPI) …
Triangle counting is a fundamental graph analytic operation that is used extensively in network science and graph mining. As the size of the graphs that needs to be analyzed continues to grow, there is a requirement in developing scalable algorithms for distributed-memory parallel systems. To this end, we present a distributed-memory triangle counting algorithm, which uses a 2D cyclic decomposition to balance the computations and reduce the communication overheads. The algorithm structures its communication and computational steps such that it reduces its memory overhead and includes key optim…
We present a novel distributed union-find algorithm that features asynchronous parallelism and k-d tree based load balancing for scalable visualization and analysis of scientific data. Applications of union-find include level set extraction and critical point tracking, but distributed union-find can suffer from high synchronization costs and imbalanced workloads across parallel processes. In this study, we prove that global synchronizations in existing distributed union-find can be eliminated without changing final results, allowing overlapped communications and computations for scalable proce…
This paper introduces a novel approach to automatic ahead-of-time (AOT) parallelization and optimization of sequential Python programs for execution on distributed heterogeneous platforms. Our approach enables AOT source-to-source transformation of Python programs, driven by the inclusion of type hints for function parameters and return values. These hints can be supplied by the programmer or obtained by dynamic profiler tools; multi-version code generation guarantees the correctness of our AOT transformation in all cases. Our compilation framework performs automatic parallelization and sophis…
Parallel Collaborative ADMM Privacy Computing and Adaptive GPU Acceleration for Distributed Edge Networks
College of Cyber Security, Jinan University（暨南大学网络空间安全学院）；Department of Electrical and Computer Engineering, Western University（韦仕敦大学电气与计算机工程系）；School of Computer Science, Guangzhou University（广州大学计算机科学与工程学院）
This paper presents the design, implementation, and evaluation of the PyTorch distributed data parallel module. PyTorch is a widely-adopted scientific computing package used in deep learning research and applications. Recent advances in deep learning argue for the value of large datasets and large models, which necessitates the ability to scale out model training to more computational resources. Data parallelism has emerged as a popular solution for distributed training thanks to its straightforward principle and broad applicability. In general, the technique of distributed data parallelism re…
Graph coloring is often used in parallelizing scientific computations that run in distributed and multi-GPU environments; it identifies sets of independent data that can be updated in parallel. Many algorithms exist for graph coloring on a single GPU or in distributed memory, but to the best of our knowledge, hybrid MPI+GPU algorithms have been unexplored until this work. We present several MPI+GPU coloring approaches based on the distributed coloring algorithms of Gebremedhin et al. and the shared-memory algorithms of Deveci et al. . The on-node parallel coloring uses implementations in Kokko…
This paper describes the architectural design as well as key implementation details of the Open Source popt4jlib library (https://githhub.org/ioannischristou/popt4jlib) that contains a fairly large number of meta-heuristic and other exact optimization algorithms parallel/distributed Java implementations. Although we report on speedup and efficiency issues on some of the algorithms in the library, our main concern is to detail the design decisions for the key parallel/distributed infrastructure built into the library, so as to make it easier for developers to develop their own parallel implemen…
Universidad Complutense de Madrid（马德里康普顿斯大学）；The MITRE Corporation（米特雷公司）
Cloud simulation environments today are largely employed to model and simulate complex systems for remote accessibility and variable capacity requirements. In this regard, scalability issues in Modeling and Simulation (M\&S) computational requirements can be tackled through the elasticity of on-demand Cloud deployment. However, implementing a high performance cloud M\&S framework following these elastic principles is not a trivial task as parallelizing and distributing existing architectures is challenging. Indeed, both the parallel and distributed M\&S developments have evolved following sepa…
Design of a Parallel and Distributed Web Search Engine
本文描述了MOSE（我自己的搜索引擎）的体系结构，一个可扩展的并行和分布式搜索引擎的Web。MOSE是专门设计来有效地利用负担得起的并行架构，如工作站集群。其模块化和可扩展的体系结构可以很容易地进行调整，以满足手头应用的带宽要求。任务并行和数据并行的方法，利用MOSE内，以提高吞吐量和有效地使用通信，存储和计算资源。我们使用一组html文档作为基准，并在三台SMP Linux PC的集群上进行了初步实验。
Parallel implementations of distributed global optimization (DGO) [13] on MP-1 and NCUBE parallel computers revealed an approximate O(n) increase in the performance of this algorithm. Therefore, the implementation of the DGO on parallel processors can remedy the only draw back of this algorithm which is the O(n2) of execution time as the number of the dimensions increase. The speed up factor of the parallel implementations of DGO is measured with respect to the sequential execution time of the identical problem on SPARC IV computer. The best speed up was achieved by the SIMD implementation of …
A viable approach for building large-scale quantum computers is to interlink small-scale quantum computers with a quantum network to create a larger distributed quantum computer. When designing quantum algorithms for such a distributed quantum computer, one can make use of the added parallelization and distribution abilities inherent in the system. An added difficulty to then overcome for distributed quantum computing is that a complex control system to orchestrate the various components is required. In this work, we aim to address these issues. We explicitly define what it means for a quantum…
Generative adversarial networks (GANs) are widely used to learn generative models. GANs consist of two networks, a generator and a discriminator, that apply adversarial learning to optimize their parameters. This article presents a parallel/distributed implementation of a cellular competitive coevolutionary method to train two populations of GANs. A distributed memory parallel implementation is proposed for execution in high performance/supercomputing centers. Efficient results are reported on addressing the generation of handwritten digits (MNIST dataset samples). Moreover, the proposed imple…
A future is a programming construct designed for concurrent and asynchronous evaluation of code, making it particularly useful for parallel processing. The future package implements the Future API for programming with futures in R. This minimal API provides sufficient constructs for implementing parallel versions of well-established, high-level map-reduce APIs. The future ecosystem supports exception handling, output and condition relaying, parallel random number generation, and automatic identification of globals lowering the threshold to parallelize code. The Future API bridges parallel fron…
LAAS-CNRS（LAAS-法国国家科学研究中心）；Université de Toulouse（图卢兹大学）；CNRS（法国国家科学研究中心）
We describe several features of parallel or distributed asynchronous iterative algorithms such as unbounded delays, possible out of order messages or flexible communication. We concentrate on the concept of macroiteration sequence which was introduced in order to study the convergence or termination of asynchronous iterations. A survey of asynchronous iterations for convex optimization problems is also presented. Finally, a new result of convergence for parallel or distributed asynchronous iterative algorithms with flexible communication for convex optimization problems and machine learning is…
A Unitychain is a novel blockchain-like structure that drastically improves transaction scalability and security while maintaining ongoing network performance, even if participating nodes are required to perform a new Distributed Key Generation procedure for security purposes. The Unitychain structure, furthermore, enables greater parallel processing by the assignment of different network node configurations for various database and compute ranges into multiple strands of blockchains that intersect, creating a multi-helix structure, which we call a Unitychain. This thereby enables the network …
