Byzantine fault-tolerant systems have been researched for more than four decades, and although shown possible early, the solutions were impractical for a long time. With PBFT the first practical solution was proposed in 1999 and spawned new research which culminated in novel applications using it today. Although the safety and liveness properties of PBFT-type protocols have been rigorously analyzed, when it comes to practical performance only empirical results - often in artificial settings - are known and imperfections on the communication channels are not specifically considered. In this work we present the first performance model for PBFT specifically considering the impact of unreliable channels and the use of different transport protocols over them. We also did extensive simulations to verify the model and to gain more insight on the impact of deployment parameters on the overall transaction time. We show that the usage of UDP can lead to significant speedup for PBFT protocols compared to TCP when tuned accordingly even over lossy channels. Finally, we compared the simulation to a real implementation and measure the benefits of a developed improvement directly. We found that the impact on the design of the network layer has been overlooked in the past but offers some additional room for improvement when it comes to practical performance. In this work we are focusing on the optimistic case with no node failures, as this is hopefully the most relevant situation.
This paper proposes the first implementation of an atomic storage tolerant to mobile Byzantine agents. Our implementation is designed for the round-based synchronous model where the set of Byzantine nodes changes from round to round. In this model we explore the feasibility of multi-writer multi-reader atomic register prone to various mobile Byzantine behaviors. We prove upper and lower bounds for solving the atomic storage in all the explored models. Our results, significantly different from the static case, advocate for a deeper study of the main building blocks of distributed computing whil…
In this paper we demonstrate how stateful Byzantine Fault Tolerant services may be hosted on a Chord ring. The strategy presented is fourfold: firstly a replication scheme that dissociates the maintenance of replicated service state from ring recovery is developed. Secondly, clients of the ring based services are made replication aware. Thirdly, a consensus protocol is introduced that supports the serialization of updates. Finally Byzantine fault tolerant replication protocols are developed that ensure the integrity of service data hosted on the ring.
It is a critical matter for a blockchain system whether a Byzantine fault tolerance (BFT) can be guaranteed during a consensus process. Can connected vehicles (CVs) achieve the BFT consensus when the vehicles keep mobile? This paper seeks an answer to this fundamental question. It focuses on characterizing the impact of mobility on the performance of a BFT consensus among CVs.
We describe an approach to modelling a Byzantine tolerant distributed algorithm as a family of related finite state machines, generated from a single meta-model. Various artefacts are generated from each state machine, including diagrams and source-level protocol implementations. The approach allows a state machine formulation to be applied to problems for which it would not otherwise be suitable, increasing confidence in correctness.
Self-stabilization is a versatile approach to fault-tolerance since it permits a distributed system to recover from any transient fault that arbitrarily corrupts the contents of all memories in the system. Byzantine tolerance is an attractive feature of distributed systems that permits to cope with arbitrary malicious behaviors. This paper focus on systems that are both self-stabilizing and Byzantine tolerant. We consider the well known problem of constructing a maximum metric tree in this context. Combining these two properties is known to induce many impossibility results. In this paper, we …
We propose three new robust aggregation rules for distributed synchronous Stochastic Gradient Descent~(SGD) under a general Byzantine failure model. The attackers can arbitrarily manipulate the data transferred between the servers and the workers in the parameter server~(PS) architecture. We prove the Byzantine resilience properties of these aggregation rules. Empirical analysis shows that the proposed techniques outperform current approaches for realistic use cases and Byzantine attack scenarios.
In this note, we revisit EZBFT[2] and present safety, liveness and execution consistency violations in the protocol. To demonstrate these violations, we present simple scenarios, involving only four replicas, two clients, and one or two owner changes. We also note shortcomings of the presented TLA+ specification used to model check the proposed protocol.
On the Solvability of Byzantine-tolerant Reliable Communication in Dynamic Networks
Sapienza University of Rome（罗马大学）；Department of Engineering, Niccolò Cusano University（尼科洛·库萨诺大学工程学院）；Sorbonne Université, CNRS, LIP6（索邦大学、法国国家科学研究中心、LIP6实验室）；Institut Universitaire de France（法兰西学院）
We present ezBFT, a novel leaderless, distributed consensus protocol capable of tolerating byzantine faults. ezBFT's main goal is to minimize the client-side latency in WAN deployments. It achieves this by (i) having no designated primary replica, and instead, enabling every replica to order the requests that it receives from clients; (ii) using only three communication steps to order requests in the common case; and (iii) involving clients actively in the consensus process. In addition, ezBFT minimizes the potentially negative effect of a byzantine replica on the overall system performance. W…
In this paper, we describe a novel proactive recovery scheme based on service migration for long-running Byzantine fault tolerant systems. Proactive recovery is an essential method for ensuring long term reliability of fault tolerant systems that are under continuous threats from malicious adversaries. The primary benefit of our proactive recovery scheme is a reduced vulnerability window. This is achieved by removing the time-consuming reboot step from the critical path of proactive recovery. Our migration-based proactive recovery is coordinated among the replicas, therefore, it can automatica…
Recently, new defense techniques have been developed to tolerate Byzantine failures for distributed machine learning. The Byzantine model captures workers that behave arbitrarily, including malicious and compromised workers. In this paper, we break two prevailing Byzantine-tolerant techniques. Specifically we show robust aggregation methods for synchronous SGD -- coordinate-wise median and Krum -- can be broken using new attack strategies based on inner product manipulation. We prove our results theoretically, as well as show empirical validation.
In this paper, we present a Byzantine fault tolerant distributed commit protocol for transactions running over untrusted networks. The traditional two-phase commit protocol is enhanced by replicating the coordinator and by running a Byzantine agreement algorithm among the coordinator replicas. Our protocol can tolerate Byzantine faults at the coordinator replicas and a subset of malicious faults at the participants. A decision certificate, which includes a set of registration records and a set of votes from participants, is used to facilitate the coordinator replicas to reach a Byzantine agree…
This paper proposes the first implementation of a self-stabilizing regular register emulated by $n$ servers that is tolerant to both mobile Byzantine agents, and \emph{transient failures} in a round-free synchronous model. Differently from existing Mobile Byzantine tolerant register implementations, this paper considers a more powerful adversary where (i) the message delay (i.e., $δ$) and the period of mobile Byzantine agents movement (i.e., $Δ$) are completely decoupled and (ii) servers are not aware of their state i.e., they do not know if they have been corrupted or not by a mobile Byzantin…
One of the most celebrated problems of fault-tolerant distributed computing is the consensus problem. It was shown to abstract a myriad of problems in which processes have to agree on a single value. Consensus applications include fundamental services for the environments of the Cloud or Blockchain. In such challenging environments, malicious behavior is often modeled as adversarial Byzantine faults. At OPODIS 2010, Moste}faoui and Raynal, in short, MR, presented a Byzantine- and intrusion-tolerant solution to consensus in which the decided value cannot be a value proposed only by Byzantine pr…
All practical applications contain some degree of nondeterminism. When such applications are replicated to achieve Byzantine fault tolerance (BFT), their nondeterministic operations must be controlled to ensure replica consistency. To the best of our knowledge, only the most simplistic types of replica nondeterminism have been dealt with. Furthermore, there lacks a systematic approach to handling common types of nondeterminism. In this paper, we propose a classification of common types of replica nondeterminism with respect to the requirement of achieving Byzantine fault tolerance, and describ…
The development of fault-tolerant distributed systems that can tolerate Byzantine behavior has traditionally been focused on consensus protocols, which support fully-replicated designs. For the development of more sophisticated high-performance Byzantine distributed systems, more specialized fault-tolerant communication primitives are necessary, however. In this paper, we identify an essential communication primitive and study it in depth. In specifics, we formalize the cluster-sending problem, the problem of sending a message from one Byzantine cluster to another Byzantine cluster in a reliab…
In this paper we propose Aleph, a leaderless, fully asynchronous, Byzantine fault tolerant consensus protocol for ordering messages exchanged among processes. It is based on a distributed construction of a partially ordered set and the algorithm for reaching a consensus on its extension to a total order. To achieve the consensus, the processes perform computations based only on a local copy of the data structure, however, they are bound to end with the same results. Our algorithm uses a dual-threshold coin-tossing scheme as a randomization strategy and establishes the agreement in an expected …
In this paper we address Approximate Agreement problem in the Mobile Byzantine faults model. Our contribution is threefold. First, we propose the the first mapping from the existing variants of Mobile Byzantine models to the Mixed-Mode faults model.This mapping further help us to prove the correctness of class MSR (Mean-Subsequence-Reduce) Approximate Agreement algorithms in the Mobile Byzantine fault model, and is of independent interest. Secondly, we prove lower bounds for solving Approximate Agreement under all existing Mobile Byzantine faults models. Interestingly, these lower bounds are d…
A shared read/write register emulation provides the illusion of shared-memory on top of message-passing models. The main hurdle with such emulations is dealing with server faults in the system. Several crash-tolerant register emulations in static systems require algorithms to replicate the value of the shared register onto a majority of servers. Majority correctness is necessary for such emulations. Byzantine faults are considered to be the worst kind of faults that can happen in any distributed system. Emulating a Byzantine-tolerant register requires replicating the register value on to more …
This report considers the problem of Byzantine fault-tolerance in synchronous parallelized learning that is founded on the parallelized stochastic gradient descent (parallelized-SGD) algorithm. The system comprises a master, and $n$ workers, where up to $f$ of the workers are Byzantine faulty. Byzantine workers need not follow the master's instructions correctly, and might send malicious incorrect (or faulty) information. The identity of the Byzantine workers remains fixed throughout the learning process, and is unknown a priori to the master. We propose two coding schemes, a deterministic sch…
We study a well-known communication abstraction called Byzantine Reliable Broadcast (BRB). This abstraction is central in the design and implementation of fault-tolerant distributed systems, as many fault-tolerant distributed applications require communication with provable guarantees on message deliveries. Our study focuses on fault-tolerant implementations for message-passing systems that are prone to process-failures, such as crashes and malicious behavior. At PODC 1983, Bracha and Toueg, in short, BT, solved the BRB problem. BT has optimal resilience since it can deal with t < n/3 Byzantin…
Cloud computing management are beyond typical human narratives. However if a virtual system is not effectively designed to tolerate Byzantine faults, it could lead to a faultily executed mission rather than a cloud crash. The cloud could recover from the crash but it could not recover from the loss of credibility. Moreover no amount of replication or fault handling measures can be helpful in facing a Byzantine fault unless the virtual system is designed to detect, tolerate and eliminate such faults. However research efforts that are made to address Byzantine faults have not provided convincing…
In this work, we consider a generalized fault model that can be used to represent a wide range of failure scenarios, including correlated failures and non-uniform node reliabilities. This fault model is general in the sense that fault models studied in prior related work, such as f -total and f -local models, are special cases of the generalized fault model. Under the generalized fault model, we explore iterative approximate Byzantine consensus (IABC) algorithms in arbitrary directed networks. We prove a necessary and sufficient condition for the existence of IABC algorithms. The use of the ge…
We explore the correctness of the Certified Propagation Algorithm (CPA) [6, 1, 8, 5] in solving broadcast with locally bounded Byzantine faults. CPA allows the nodes to use only local information regarding the network topology. We provide a tight necessary and sufficient condition on the network topology for the correctness of CPA. To the best of our knowledge, this work is the first to solve the open problem in [8]. We also present some simple extensions of this result
Byzantine Agreement introduced in [Pease, Shostak, Lamport, 80] is a widely used building block of reliable distributed protocols. It simulates broadcast despite the presence of faulty parties within the network, traditionally using only private unicast links. Under such conditions, Byzantine Agreement requires more than 2/3 of the parties to be compliant. [Fitzi, Maurer, 00], constructed a Byzantine Agreement protocol for any compliant majority based on an additional primitive allowing transmission to any two parties simultaneously. They proposed a problem of generalizing these results to wid…
In this note, we observe a safety violation in Zyzzyva and a liveness violation in FaB. To demonstrate these issues, we require relatively simple scenarios, involving only four replicas, and one or two view changes. In all of them, the problem is manifested already in the first log slot.
Byzantine fault tolerance (BFT) has been extensively studied in distributed trustless systems to guarantee system's functioning when up to 1/3 Byzantine processes exist. Despite a plethora of previous work in BFT systems, they are mainly concerned about common knowledge deducible from the states of all participant processes. In BFT systems, it is crucial to know about which knowledge a process knows about the states of other processes and the global state of the system. However, there is a lack of studies about common knowledge of Byzantine faults, such as, whether existence of a Byzantine nod…
We introduce an automated parameterized verification method for fault-tolerant distributed algorithms (FTDA). FTDAs are parameterized by both the number of processes and the assumed maximum number of Byzantine faulty processes. At the center of our technique is a parametric interval abstraction (PIA) where the interval boundaries are arithmetic expressions over parameters. Using PIA for both data abstraction and a new form of counter abstraction, we reduce the parameterized problem to finite-state model checking. We demonstrate the practical feasibility of our method by verifying several varia…
We consider the problem of Byzantine fault-tolerance in the peer-to-peer (P2P) distributed gradient-descent method -- a prominent algorithm for distributed optimization in a P2P system. In this problem, the system comprises of multiple agents, and each agent has a local cost function. In the fault-free case, when all the agents are honest, the P2P distributed gradient-descent method allows all the agents to reach a consensus on a solution that minimizes their aggregate cost. However, we consider a scenario where a certain number of agents may be Byzantine faulty. Such faulty agents may not fol…
In this paper we analyse the correctness of Istanbul BFT (IBFT), which is a Byzantine-fault-tolerant (BFT) proof-of-authority (PoA) blockchain consensus protocol that ensures immediate finality. We show that the IBFT protocol does not guarantee Byzantine-fault-tolerant consistency and liveness when operating in an eventually synchronous network, and we propose modifications to the protocol to ensure both Byzantine-fault-tolerant consistency and liveness in eventually synchronous settings.
Causal consistency is a widely used weak consistency model that allows high availability despite network partitions. There are plenty of research prototypes and industrial deployments of causally consistent distributed systems. However, as far as we know, none of them consider Byzantine faults, except Byz-RCM proposed by Tseng et al. Byz-RCM achieves causal consistency in the client-server model with $3f + 1$ servers where up to $f$ servers may suffer Byzantine faults, but assumes that clients are non-Byzantine. In this work, we present Byz-Gentlerain, the first causal consistency protocol whi…
Strong replica consistency is often achieved by writing deterministic applications, or by using a variety of mechanisms to render replicas deterministic. There exists a large body of work on how to render replicas deterministic under the benign fault model. However, when replicas can be subject to malicious faults, most of the previous work is no longer effective. Furthermore, the determinism of the replicas is often considered harmful from the security perspective and for many applications, their integrity strongly depends on the randomness of some of their internal operations. This calls for…
Among those faults Byzantine faults offers serious challenge to fault tolerance mechanism, because it often go undetected at the initial stage and it can easily propagate to other VMs before a detection is made. Consequently some of the mission critical application such as air traffic control, online baking etc still staying away from the cloud for such reasons. However if a Byzantine faults is not detected and tolerated at initial stage then applications such as big data analytics can go completely wrong in spite of hours of computations performed by the entire cloud. Therefore in the previou…
Byzantine-Fault-Tolerant (BFT) systems are rapidly emerging as a viable technology for production-grade systems, notably in closed consortia deployments for nancial and supply-chain applications. Unfortunately, most algorithms proposed so far to coordinate these systems suffer from substantial scalability issues, and lack important features to implement Internet-scale governance mechanisms. In this paper, we observe that many application workloads offer little concurrency, and propose PnyxDB, an eventually-consistent Byzantine Fault Tolerant replicated datastore that exhibits both high scalabi…
Fault-tolerant consensus has been studied extensively in the literature, because it is one of the most important distributed primitives and has wide applications in practice. This paper surveys important results on fault-tolerant consensus in message-passing networks, and the focus is on results from the past decade. Particularly, we categorize the results into two groups: new problem formulations and practical applications. In the first part, we discuss new ways to define the consensus problem, which includes larger input domains, link fault models, different network models . . . etc, and bri…
Hyperledger Fabric (HLF) is a flexible permissioned blockchain platform designed for business applications beyond the basic digital coin addressed by Bitcoin and other existing networks. A key property of HLF is its extensibility, and in particular the support for multiple ordering services for building the blockchain. Nonetheless, the version 1.0 was launched in early 2017 without an implementation of a Byzantine fault-tolerant (BFT) ordering service. To overcome this limitation, we designed, implemented, and evaluated a BFT ordering service for HLF on top of the BFT-SMaRt state machine repli…
Byzantine fault tolerant (BFT) state machine replication (SMR) is an important building block for constructing permissioned blockchain systems. In contrast to Nakamoto Consensus where any block obtains higher assurance as buried deeper in the blockchain, in BFT SMR, any committed block is secure has a fixed resilience threshold. In this paper, we investigate strengthened fault tolerance (SFT) in BFT SMR under partial synchrony, which provides gradually increased resilience guarantees (like Nakamoto Consensus) during an optimistic period when the network is synchronous and the number of Byzanti…
FaB Paxos[5] sets a lower bound of 5f + 1 replicas for any two-step consensus protocols tolerating f byzantine failures. Yet, hBFT[3] promises a two-step consensus protocol with only 3f + 1 replicas. As a result, it violates safety property of a consensus protocol. In this note, we review the lower bound set by FaB Paxos and present a simple execution scenario that produces a safety violation in hBFT. To demonstrate the scenario, we require a relatively simple setup with only 4 replicas and one view-change.
We propose a framework to build formal developments for robot networks using the COQ proof assistant, to state and to prove formally various properties. We focus in this paper on impossibility proofs, as it is natural to take advantage of the COQ higher order calculus to reason about algorithms as abstract objects. We present in particular formal proofs of two impossibility results forconvergence of oblivious mobile robots if respectively more than one half and more than one third of the robots exhibit Byzantine failures, starting from the original theorems by Bouzid et al.. Thanks to our form…
In a previous note (arXiv:1712.01367 [cs.DC]) , we observed a safety violation in Zyzzyva and a liveness violation in FaB. In this manuscript, we sketch fixes to both. The same view-change core is applied in the two schemes, and additionally, applied to combine them and create a single, enhanced scheme that has the benefits of both approaches.
This paper presents DuoBFT, a Byzantine fault-tolerant protocol that uses trusted components to provide commit decisions in the Hybrid fault model in addition to commit decisions in the BFT model. By doing so, it enables the clients to choose the response fault model for its commands. Internally, DuoBFT commits each client command under both the hybrid and Byzantine models, but since hybrid commits take fewer communication steps and use smaller quorums than BFT commits, clients can benefit from the low-latency commits in the hybrid model. DuoBFT uses a common view-change change protocol to han…
We present concept and implementation of a self-stabilizing Byzantine fault-tolerant distributed clock generation scheme for multi-synchronous GALS architectures in critical applications. It combines a variant of a recently introduced self-stabilizing algorithm for generating low-frequency, low-accuracy synchronized pulses with a simple non-stabilizing high-frequency, high-accuracy clock synchronization algorithm. We provide thorough correctness proofs and a performance analysis, which use methods from fault-tolerant distributed computing research but also addresses hardware-related issues lik…
A set of mobile robots is placed at points of an infinite line. The robots are equipped with GPS devices and they may communicate their positions on the line to a central authority. The collection contains an unknown subset of "spies", i.e., byzantine robots, which are indistinguishable from the non-faulty ones. The set of the non-faulty robots need to rendezvous in the shortest possible time in order to perform some task, while the byzantine robots may try to delay their rendezvous for as long as possible. The problem facing a central authority is to determine trajectories for all robots so a…
It is a common belief that Byzantine fault-tolerant solutions for consensus are significantly slower than their crash fault-tolerant counterparts. Indeed, in PBFT, the most widely known Byzantine fault-tolerant consensus protocol, it takes three message delays to decide a value, in contrast with just two in Paxos. This motivates the search for fast Byzantine consensus algorithms that can produce decisions after just two message delays \emph{in the common case}, e.g., under the assumption that the current leader is correct and not suspected by correct processes. The (optimal) two-step latency c…
