最近，空闲机制已被引入到分布式\emph{first order}方法的上下文中，用于最小化通用连接网络上的节点的局部凸成本之和。利用空闲机制，每个节点$i$在每次迭代$k$处是活动的--更新其解估计并与其网络邻居交换消息--概率为$p_k$，并且它以概率$1-p_k$保持空闲，而激活在节点之间和迭代之间都是独立的。在本文中，我们证明了空转机制可以成功地纳入\emph{distributed second order methods}也。具体来说，我们应用闲置机制，最近提出的分布式拟牛顿法（DQN）。我们首先从理论上表明，当$p_k$增长到一个在一个受控的方式跨越迭代，DQN与空闲表现出非常相似的理论收敛性和收敛速度的属性作为标准的DQN方法，从而实现了相同的顺序的收敛速度（R-线性）作为标准的DQN，但显着便宜的更新。仿真实例证实了将空转机制的好处，证明了该方法的灵活性方面的选择的$p_k$的，并比较所提出的空转方法与相关算法从文献。
英文摘要：
Recently, an idling mechanism has been introduced in the context of distributed \emph{first order} methods for minimization of a sum of nodes' local convex costs over a generic, connected network. With the idling mechanism, each node $i$, at each iteration $k$, is active -- updates its solution estimate and exchanges messages with its network neighborhood -- with probability $p_k$, and it stays idle with probability $1-p_k$, while the activations are independent both across nodes and across iterations. In this paper, we demonstrate that the idling mechanism can be successfully incorporated in \emph{distributed second order methods} also. Specifically, we apply the idling mechanism to the recently proposed Distributed Quasi Newton method (DQN). We first show theoretically that, when $p_k$ grows to one across iterations in a controlled manner, DQN with idling exhibits very similar theoretical convergence and convergence rates properties as the standard DQN method, thus achieving the same order of convergence rate (R-linear) as the standard DQN, but with significantly cheaper updates. Simulation examples confirm the benefits of incorporating the idling mechanism, demonstrate the method's flexibility with respect to the choice of the $p_k$'s, and compare the proposed idling method with related algorithms from the literature.
We consider a MapReduce-type task running in a distributed computing model which consists of ${K}$ edge computing nodes distributed across the edge of the network and a Master node that assists the edge nodes to compute output functions. The Master node and the edge nodes, both equipped with some storage memories and computing capabilities, are connected through a multicast network. We define the communication time spent during the transmission for the sequential implementation (all nodes send symbols sequentially) and parallel implementation (the Master node can send symbols during the edge n…
In distributed storage systems (DSSs), the optimal tradeoff between node storage and repair bandwidth is an important issue for designing distributed coding strategies to ensure large scale data reliability. The capacity of DSSs is obtained as a function of node storage and repair bandwidth parameters, characterizing the tradeoff. There are lots of works on DSSs with clusters (racks) where the repair bandwidths from intra-cluster and cross-cluster are differentiated. However, separate nodes are also prevalent in the realistic DSSs, but the works on DSSs with clusters and separate nodes (CSN-DS…
Distributed computation is a framework used to break down a complex computational task into smaller tasks and distributing them among computational nodes. Erasure correction codes have recently been introduced and have become a popular workaround to the well known ``straggling nodes'' problem, in particular, by matching linear coding for linear computation tasks. It was observed that decoding tends to amplify the computation ``noise'', i.e., the numerical errors at the computation nodes. We propose taking advantage of the case that more nodes return than minimally required. We show how a cleve…
The majority of works in distributed storage networks assume a simple network model with a collection of identical storage nodes with the same communication cost between the nodes. In this paper, we consider a realistic multi-rack distributed data storage network and present a code design framework for this model. Considering the cheaper data transmission within the racks, our code construction method is able to locally repair the nodes failure within the same rack by using only the survived nodes in the same rack. However, in the case of severe failure patterns when the information content of…
Distributed databases often suffer unequal distribution of data among storage nodes, which is known as `data skew'. Data skew arises from a number of causes such as removal of existing storage nodes and addition of new empty nodes to the database. Data skew leads to performance degradations and \textcolor{black}{thus} necessitates `rebalancing' at regular intervals to reduce the amount of skew. We define an $r$-balanced distributed database as a distributed database in which the storage across the nodes has uniform size, and each bit of the data is replicated in $r$ distinct storage nodes. We …
In this paper, we study the problem of distributed multi-user secret sharing, including a trusted master node, $N\in \mathbb{N}$ storage nodes, and $K$ users, where each user has access to the contents of a subset of storage nodes. Each user has an independent secret message with certain rate, defined as the size of the message normalized by the size of a storage node. Having access to the secret messages, the trusted master node places encoded shares in the storage nodes, such that (i) each user can recover its own message from the content of the storage nodes that it has access to, (ii) each…
The performance of replication-based distributed databases is affected due to non-uniform storage across storage nodes (also called \textit{data skew}) and reduction in the replication factor during operation, particularly due to node additions or removals. Data rebalancing refers to the communication involved between the nodes in correcting this data skew, while maintaining the replication factor. For carefully designed distributed databases, transmitting coded symbols during the rebalancing phase has been recently shown to reduce the communication load of rebalancing. In this work, we look a…
An $(n, k, d, α, β, M)$-ERRC (exact-repair regenerating code) is a collection of $n$ nodes used to store a file. For a file of total size $M$, each node stores $α$ symbols, any $k$ nodes recover the file, and any $d$ nodes repair any other node via sending out $β$ symbols. We establish a multilinear algebra foundation to assemble $(n, k, d, α, β, M)$-ERRCs for all meaningful $(n, k, d)$ tuples. Our ERRCs tie the $α/M$-versus-$β/M$ trade-off with cascade codes, the best known construction for this trade-off. We give directions on how these ERRCs repair multiple failures.
In this paper, we study a distributed learning problem constrained by constant communication bits. Specifically, we consider the distributed hypothesis testing (DHT) problem where two distributed nodes are constrained to transmit a constant number of bits to a central decoder. In such cases, we show that in order to achieve the optimal error exponents, it suffices to consider the empirical distributions of observed data sequences and encode them to the transmission bits. With such a coding strategy, we develop a geometric approach in the distribution spaces and establish an inner bound of erro…
We consider a distributed storage system with $n$ nodes, where a user can recover the stored file from any $k$ nodes, and study the problem of repairing $r$ partially failed nodes. We consider \textit{broadcast repair}, that is, $d$ surviving nodes transmit broadcast messages on an error-free wireless channel to the $r$ nodes being repaired, which are then used, together with the surviving data in the local memories of the failed nodes, to recover the lost content. First, we derive the trade-off between the storage capacity and the repair bandwidth for partial repair of multiple failed nodes, …
Large-scale machine learning and data mining applications require computer systems to perform massive matrix-vector and matrix-matrix multiplication operations that need to be parallelized across multiple nodes. The presence of straggling nodes -- computing nodes that unpredictably slowdown or fail -- is a major bottleneck in such distributed computations. Ideal load balancing strategies that dynamically allocate more tasks to faster nodes require knowledge or monitoring of node speeds as well as the ability to quickly move data. Recently proposed fixed-rate erasure coding strategies can handl…
We consider a status update system in which the update packets need to be processed to extract the embedded useful information. The source node sends the acquired information to a computation unit (CU) which consists of a master node and $n$ worker nodes. The master node distributes the received computation task to the worker nodes. Upon computation, the master node aggregates the results and sends them back to the source node to keep it \emph{updated}. We investigate the age performance of uncoded and coded (repetition coded, MDS coded, and multi-message MDS (MM-MDS) coded) schemes in the pre…
