Multi-level Forwarding and Scheduling Recovery Algorithm in Rapidly-changing Network for Erasure-coded Clusters
Hai Zhou, Dan Feng, Yuchong Hu
备注：
We have modified the algorithm of this article, and a submission meeting is required. However, the meeting request cannot be published, so we apply to withdraw the article
A key design goal of erasure-coded clusters is to reduce the repair time. The existing Erasure-coded data repair schemes are roughly classified into two categories: 1. Designing rapid data repair (e.g., PPR) in a homogeneous environment. 2. Constructing data repair (e.g., PPT) based on bandwidth in a heterogeneous environment. However, these solutions are difficult to cope with the heterogeneous and Rapidly-changing network in erasure-coded clusters. To address this problem, a bandwidth-aware multi-level forwarding repair algorithm, called BMFRepair, is proposed. BMFRepair monitors the network bandwidth in real time when data is forwarded, and selects idle nodes with high-bandwidth links to assist in forwarding. Thus, it can reduce the time bottleneck caused by low link transmission. At the same time, multi-node repair becomes very complicated when the bandwidth changes drastically. A multi-node scheduling repairing algorithm, called MSRepair, is proposed for multi-node repairing problems, which can repair multiple failed blocks in parallel by scheduling node resources. The two algorithms can flexibly adapt to the rapidly changing network environment and make full use of the bandwidth resources of idle nodes. Most importantly, algorithms can continuously adjust the repair plan according to the bandwidth change in fast and dynamic network. The algorithms have been evaluated by both simulations on Mininet and real experiments on Aliyun cloud platform ECS. Results show that compared with the state-of-the-art repair schemes PPR and PPT, the algorithms can significantly reduce the repair time in rapidly-changing network.
To enhance the reliability of LoRa sensor networks, we propose two opportunistic forwarding protocols in which relays overhear sensor-to-gateway messages and periodically forward them to a gateway. To reduce the number of transmissions, the relays forward XOR sums of the overheard messages. One protocol employs a single relay node, while two relays cooperate in the other. With a single relay, our proposed protocol requires fewer relay transmissions to achieve similar loss rates as conventional uncoded forwarding in which every received message is immediately forwarded. We demonstrate up to 55%…
As consumers are increasingly engaged in social networking and E-commerce activities, businesses grow to rely on Big Data analytics for intelligence, and traditional IT infrastructures continue to migrate to the cloud and edge, these trends cause distributed data storage demand to rise at an unprecedented speed. Erasure coding has seen itself quickly emerged as a promising technique to reduce storage cost while providing similar reliability as replicated systems, widely adopted by companies like Facebook, Microsoft and Google. However, it also brings new challenges in characterizing and optimi…
Efficient resource discovery and availability improvement are very important issues in unstructured P2P networks. In this paper, a bio-inspired resource discovery scheme inspired by the principle of elephants migration is proposed. A replication scheme based on Q-learning and erasure codes is also introduced. Simulation results show that the proposed schemes significantly increases query success rate and availability, and reduces the network traffic as the resources are effectively distributed to well-performing nodes.
Distributed storage systems are known to be susceptible to long tails in response time. In modern online storage systems such as Bing, Facebook, and Amazon, the long tails of the service latency are of particular concern. with 99.9th percentile response times being orders of magnitude worse than the mean. As erasure codes emerge as a popular technique to achieve high data reliability in distributed storage while attaining space efficiency, taming tail latency still remains an open problem due to the lack of mathematical models for analyzing such systems. To this end, we propose a framework for…
Erasure codes, such as Reed-Solomon (RS) codes, are being increasingly employed in data centers to combat the cost of reliably storing large amounts of data. Although these codes provide optimal storage efficiency, they require significantly high network and disk usage during recovery of missing data. In this paper, we first present a study on the impact of recovery operations of erasure-coded data on the data-center network, based on measurements from Facebook's warehouse cluster in production. To the best of our knowledge, this is the first study of its kind available in the literature. Our …
The demand for global video has been burgeoning across industries. With the expansion and improvement of video-streaming services, cloud-based video is evolving into a necessary feature of any successful business for reaching internal and external audiences. This paper considers video streaming over distributed systems where the video segments are encoded using an erasure code for better reliability thus being the first work to our best knowledge that considers video streaming over erasure-coded distributed cloud systems. The download time of each coded chunk of each video segment is character…
We consider an energy harvesting transmitter sending status updates to a receiver over an erasure channel, where each status update is of length $k$ symbols. The energy arrivals and the channel erasures are independent and identically distributed (i.i.d.) and Bernoulli distributed in each slot. In order to combat the effects of the erasures in the channel and the uncertainty in the energy arrivals, we use channel coding to encode the status update symbols. We consider two types of channel coding: maximum distance separable (MDS) codes and rateless erasure codes. For each of these models, we st…
Erasure codes have been widely considered a promising solution to enhance data reliability at low storage costs. However, in modern geo-distributed storage systems, erasure codes may incur high data access latency as they require data retrieval from multiple remote storage nodes. This hinders the extensive application of erasure codes to data-intensive applications. This paper proposes novel caching schemes to achieve low latency in distributed coded storage systems. Experiments based on Amazon Simple Storage Service confirm the positive correlation between the latency and the physical distanc…
We analyze the performance of TCP and TCP with network coding (TCP/NC) in lossy networks. We build upon the framework introduced by Padhye et al. and characterize the throughput behavior of classical TCP and TCP/NC as a function of erasure probability, round-trip time, maximum window size, and duration of the connection. Our analytical results show that network coding masks random erasures from TCP, thus preventing TCP's performance degradation in lossy networks. It is further seen that TCP/NC has significant throughput gains over TCP. In addition, we show that TCP/NC may lead to cost reductio…
We analyze the performance of TCP and TCP with network coding (TCP/NC) in lossy wireless networks. We build upon the simple framework introduced by Padhye et al. and characterize the throughput behavior of classical TCP as well as TCP/NC as a function of erasure rate, round-trip time, maximum window size, and duration of the connection. Our analytical results show that network coding masks erasures and losses from TCP, thus preventing TCP's performance degradation in lossy networks, such as wireless networks. It is further seen that TCP/NC has significant throughput gains over TCP. In addition…
This study explores the throughput and delay that can be achieved by various forwarding schemes employing multiple paths and different degrees of redundancy focusing on linear network coding. The key contribution of the study is an analytical framework for modeling the throughput and delay for various schemes considering wireless mesh networks where, unicast traffic is forwarded and hop-by-hop retransmissions are employed for achieving reliability. The analytical framework is generalized for an arbitrary number of paths and hops per path. Another key contribution of the study is the evaluation…
This paper investigates the use of redundancy and self repairing against node failures in distributed storage systems, using various strategies. In replication method, access to one replication node is sufficient to reconstruct a lost node, while in MDS erasure coded systems which are optimal in terms of redundancy-reliability tradeoff, a single node failure is repaired after recovering the entire stored data. Moreover, regenerating codes yield a tradeoff curve between storage capacity and repair bandwidth. The current paper aims at investigating a new storage code. Specifically, we propose a …
We propose application-layer coding schemes to recover lost data in delay-sensitive uplink (sensor-to-gateway) communications in the Internet of Things. Built on an approach that combines retransmissions and forward erasure correction, the proposed schemes' salient features include low computational complexity and the ability to exploit sporadic receiver feedback for efficient data recovery. Reduced complexity is achieved by keeping the number of coded transmissions as low as possible and by devising a mechanism to compute the optimal degree of a coded packet in O(1). Our major contributions a…
We define a class of multi--hop erasure networks that approximates a wireless multi--hop network. The network carries unicast flows for multiple users, and each information packet within a flow is required to be decoded at the flow destination within a specified delay deadline. The allocation of coding rates amongst flows/users is constrained by network capacity. We propose a proportional fair transmission scheme that maximises the sum utility of flow throughputs. This is achieved by {\em jointly optimising the packet coding rates and the allocation of bits of coded packets across transmission…
Revisiting the Interface between Error and Erasure Correction in Wireless Standards
Massachusetts Institute of Technology（麻省理工学院）；Nokia Bell Labs（诺基亚贝尔实验室）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Technion-Israel Institute of Technology（以色列理工学院）
We consider a geographically constrained caching community where popular data files are cached on mobile terminals and distributed through Device-to-Device (D2D) communications. Further, to ensure availability, data files are protected against user mobility, or churn, with erasure coding. Communication and storage costs (in units of energy) are considered. We focus on finding the coding method that minimizes the overall cost in the network. Closed-form expressions for the expected energy consumption incurred by data delivery and redundancy maintenance are derived, and it is shown that coding s…
In this paper, we study Coded relay (Crelay) in multi-hop wireless networks. Crelay exploits both partial packets and overhearing capabilities of the wireless nodes, and uses Forward Error Correction code in packet forwarding. When a node overhears a partial packet from an upstream node, it informs the upstream node about the number of parity bytes needed to correct the errors, such that the upstream node need only send a small amount of parity bytes instead of the complete packet, hence improving the network efficiency. Our main contributions include the following. First, we propose an effici…
Current medium access control mechanisms are based on collision avoidance and collided packets are discarded. The recent work on ZigZag decoding departs from this approach by recovering the original packets from multiple collisions. In this paper, we present an algebraic representation of collisions which allows us to view each collision as a linear combination of the original packets. The transmitted, colliding packets may themselves be a coded version of the original packets. We propose a new acknowledgment (ACK) mechanism for collisions based on the idea that if a set of packets collide, th…
High-throughput blockchains require efficient transaction broadcast mechanisms that can deliver transactions to most network nodes with low bandwidth overhead and latency. Existing schemes coordinate transmissions across peers to avoid sending redundant data, but they either incur a high latency or are not robust against adversarial network nodes. We present Strokkur, a new transaction broadcasting mechanism that provides both low bandwidth overhead and low latency. The core idea behind Strokkur is to avoid explicit coordination through randomized transaction coding. Rather than forward indivi…
We design and validate new block propagation protocols for the peer-to-peer (P2P) network of the Bitcoin blockchain. Despite its strong protection for security and privacy, the current Bitcoin blockchain can only support a low number of transactions per second (TPS). In this work, we redesign the current Bitcoin's networking protocol to increase TPS without changing vital components in its consensus-building protocol. In particular, we improve the compact-block relaying protocol to enable the propagation of blocks containing a massive number of transactions without inducing extra propagation l…
In this paper, we consider the issue of throughput and packet drop rate (PDR) optimization as two performance metrics for delay sensitive applications in network coded time division duplex (TDD) satellite systems with large round trip times (RTT). We adopt random linear network coding (RLNC) under two different scenarios, feedback-less and with feedback, and our goal is to jointly optimize the mean throughputs and PDRs of users in the system. For this purpose, we propose a systematic framework and start with formulating and optimizing these performance metrics for the single-user case. This fr…
Large-scale distributed storage systems typically use erasure codes to provide durability of data in the face of failures. A set of $k$ blocks to be stored is encoded using an $[n, k]$ code to generate $n$ blocks that are then stored on different storage nodes. The redundancy configuration is chosen based on the failure rates of storage devices, and is typically kept constant. However, a recent work by Kadekodi et al. shows that the failure rate of storage devices vary significantly over time, and that adapting the redundancy configuration in response to such variations provides significant be…
Network coding is an effective idea to boost the capacity of wireless networks, and a variety of studies have explored its advantages in different scenarios. However, there is not much analytical study on throughput and end-to-end delay of network coding in multi-hop wireless networks considering the specifications of IEEE 802.11 Distributed Coordination Function. In this paper, we utilize queuing theory to propose an analytical framework for bidirectional unicast flows in multi-hop wireless mesh networks. We study the throughput and end-to-end delay of inter-flow network coding under the IEEE…
Physical-layer Network Coding (PNC) can significantly improve the throughput of two-way relay channels. An interesting variant of PNC is Analog Network Coding (ANC). Almost all ANC schemes proposed to date, however, operate in a symbol by symbol manner (memoryless) and cannot exploit the redundant information in channel-coded packets to enhance performance. This paper proposes a non-memoryless ANC scheme. In particular, we design a soft-input soft-output decoder for the relay node to process the superimposed packets from the two end nodes to yield an estimated MMSE packet for forwarding back t…
This paper considers the multiple-access relay channel in a setting where two source nodes transmit packets to a destination node, both directly and via a relay node, over packet erasure channels. Intra-session network coding is used at the source nodes and inter-session network coding is employed at the relay node to combine the recovered source packets of both source nodes. In this work, we investigate the performance of the network-coded system in terms of the probability that the destination node will successfully recover the source packets of the two source nodes. We build our analysis on…
