Approximate Byzantine Fault-Tolerance in Distributed Optimization
Shuo Liu, Nirupam Gupta, Nitin H. Vaidya
备注：
43 pages, 5 figures, and 1 table. The report is an important extension to prior work https://dl.acm.org/doi/abs/10.1145/3382734.3405748, and arXiv:2003.09675; Added an alternative result with a better analysis
本文研究分布式多智能体优化中的拜占庭容错问题。在该问题中，每个智能体都有一个局部代价函数；在无故障情况下，目标是设计一种分布式算法，使所有智能体都能找到所有智能体聚合代价函数的最小点。我们考虑某些智能体可能发生拜占庭故障的情形，这种故障会使计算所有智能体聚合代价最小点这一原始目标失去意义。在这种情形下，对算法而言更合理的目标是让所有非故障智能体仅计算非故障智能体聚合代价的最小点。先前工作表明，如果 n 个智能体中至多有 f 个拜占庭智能体，那么当且仅当非故障智能体的代价满足某种称为 2f-冗余的冗余性质时，才能精确计算非故障智能体聚合代价的最小点。然而，2f-冗余是一种理想性质，只有在不存在噪声或不确定性的系统中才能满足，这会使精确容错目标在某些应用中无法实现。因此，我们引入 (f,ε)-弹性的概念，它是精确容错的推广，其目标是以 ε 精度找到非故障聚合代价的近似最小点。与 2f-冗余相比，这种近似容错可以在更弱、实践中更容易满足的条件下实现。我们得到了实现 (f,ε)-弹性的充要条件，刻画了冗余松弛与容错近似之间的关联。当智能体的代价函数可微时，我们还得到了配备鲁棒梯度聚合的分布式梯度下降方法具备 (f,ε)-弹性的条件。
英文摘要：
This paper considers the problem of Byzantine fault-tolerance in distributed multi-agent optimization. In this problem, each agent has a local cost function, and in the fault-free case, the goal is to design a distributed algorithm that allows all the agents to find a minimum point of all the agents' aggregate cost function. We consider a scenario where some agents might be Byzantine faulty that renders the original goal of computing a minimum point of all the agents' aggregate cost vacuous. A more reasonable objective for an algorithm in this scenario is to allow all the non-faulty agents to compute the minimum point of only the non-faulty agents' aggregate cost. Prior work shows that if there are up to $f$ (out of $n$) Byzantine agents then a minimum point of the non-faulty agents' aggregate cost can be computed exactly if and only if the non-faulty agents' costs satisfy a certain redundancy property called $2f$-redundancy. However, $2f$-redundancy is an ideal property that can be satisfied only in systems free from noise or uncertainties, which can make the goal of exact fault-tolerance unachievable in some applications. Thus, we introduce the notion of $(f,ε)$-resilience, a generalization of exact fault-tolerance wherein the objective is to find an approximate minimum point of the non-faulty aggregate cost, with $ε$ accuracy. This approximate fault-tolerance can be achieved under a weaker condition that is easier to satisfy in practice, compared to $2f$-redundancy. We obtain necessary and sufficient conditions for achieving $(f,ε)$-resilience characterizing the correlation between relaxation in redundancy and approximation in resilience. In case when the agents' cost functions are differentiable, we obtain conditions for $(f,ε)$-resilience of the distributed gradient-descent method when equipped with robust gradient aggregation.
In this note, we observe a safety violation in Zyzzyva and a liveness violation in FaB. To demonstrate these issues, we require relatively simple scenarios, involving only four replicas, and one or two view changes. In all of them, the problem is manifested already in the first log slot.
FaB Paxos[5] sets a lower bound of 5f + 1 replicas for any two-step consensus protocols tolerating f byzantine failures. Yet, hBFT[3] promises a two-step consensus protocol with only 3f + 1 replicas. As a result, it violates safety property of a consensus protocol. In this note, we review the lower bound set by FaB Paxos and present a simple execution scenario that produces a safety violation in hBFT. To demonstrate the scenario, we require a relatively simple setup with only 4 replicas and one view-change.
In a previous note (arXiv:1712.01367 [cs.DC]) , we observed a safety violation in Zyzzyva and a liveness violation in FaB. In this manuscript, we sketch fixes to both. The same view-change core is applied in the two schemes, and additionally, applied to combine them and create a single, enhanced scheme that has the benefits of both approaches.
Byzantine fault tolerance (BFT) has been extensively studied in distributed trustless systems to guarantee system's functioning when up to 1/3 Byzantine processes exist. Despite a plethora of previous work in BFT systems, they are mainly concerned about common knowledge deducible from the states of all participant processes. In BFT systems, it is crucial to know about which knowledge a process knows about the states of other processes and the global state of the system. However, there is a lack of studies about common knowledge of Byzantine faults, such as, whether existence of a Byzantine nod…
It is a critical matter for a blockchain system whether a Byzantine fault tolerance (BFT) can be guaranteed during a consensus process. Can connected vehicles (CVs) achieve the BFT consensus when the vehicles keep mobile? This paper seeks an answer to this fundamental question. It focuses on characterizing the impact of mobility on the performance of a BFT consensus among CVs.
We consider the problem of Byzantine fault-tolerance in the peer-to-peer (P2P) distributed gradient-descent method -- a prominent algorithm for distributed optimization in a P2P system. In this problem, the system comprises of multiple agents, and each agent has a local cost function. In the fault-free case, when all the agents are honest, the P2P distributed gradient-descent method allows all the agents to reach a consensus on a solution that minimizes their aggregate cost. However, we consider a scenario where a certain number of agents may be Byzantine faulty. Such faulty agents may not fol…
Cloud computing management are beyond typical human narratives. However if a virtual system is not effectively designed to tolerate Byzantine faults, it could lead to a faultily executed mission rather than a cloud crash. The cloud could recover from the crash but it could not recover from the loss of credibility. Moreover no amount of replication or fault handling measures can be helpful in facing a Byzantine fault unless the virtual system is designed to detect, tolerate and eliminate such faults. However research efforts that are made to address Byzantine faults have not provided convincing…
Among those faults Byzantine faults offers serious challenge to fault tolerance mechanism, because it often go undetected at the initial stage and it can easily propagate to other VMs before a detection is made. Consequently some of the mission critical application such as air traffic control, online baking etc still staying away from the cloud for such reasons. However if a Byzantine faults is not detected and tolerated at initial stage then applications such as big data analytics can go completely wrong in spite of hours of computations performed by the entire cloud. Therefore in the previou…
Byzantine fault-tolerant agreement (BFT) in a partially synchronous system usually requires 3f + 1 nodes to tolerate f faulty replicas. Due to their high throughput and finality property BFT algorithms build the core of recent permissioned blockchains. As a complex and resource-demanding infrastructure, multiple cloud providers have started offering Blockchain-as-a-Service. This eases the deployment of permissioned blockchains but places the cloud provider in a central controlling position, thereby questioning blockchains' fault tolerance and decentralization properties and their underlying BF…
This report considers the problem of Byzantine fault-tolerance in synchronous parallelized learning that is founded on the parallelized stochastic gradient descent (parallelized-SGD) algorithm. The system comprises a master, and $n$ workers, where up to $f$ of the workers are Byzantine faulty. Byzantine workers need not follow the master's instructions correctly, and might send malicious incorrect (or faulty) information. The identity of the Byzantine workers remains fixed throughout the learning process, and is unknown a priori to the master. We propose two coding schemes, a deterministic sch…
This paper describes BigBFT, a multi-leader Byzantine fault tolerance protocol that achieves high throughput and scalable consensus in blockchain systems. BigBFT achieves this by (1) enabling every node to be a leader that can propose and order the blocks in parallel, (2) piggybacking votes within rounds, (3) pipelining blocks across rounds, and (4) using only two communication steps to order blocks in the common case. BigBFT has an amortized communication cost of $O(n)$ over $n$ requests. We evaluate BigBFT's performance both analytically, using back-of-the-envelope load formulas to construct…
With the rapid development of blockchain, Byzantine fault-tolerant protocols have attracted revived interest recently. To overcome the theoretical bounds of Byzantine fault tolerance, many protocols attempt to use Trusted Execution Environment (TEE) to prevent equivocation and improve performance and scalability. However, due to the broken quorum intersection assumption caused by the reduction of the replica number, the improvement is mostly at the cost of increased communication complexity which prevents existing TEE-based protocols to be applied to large-scale blockchain systems. In this pap…
In this note, we revisit EZBFT[2] and present safety, liveness and execution consistency violations in the protocol. To demonstrate these violations, we present simple scenarios, involving only four replicas, two clients, and one or two owner changes. We also note shortcomings of the presented TLA+ specification used to model check the proposed protocol.
We present ezBFT, a novel leaderless, distributed consensus protocol capable of tolerating byzantine faults. ezBFT's main goal is to minimize the client-side latency in WAN deployments. It achieves this by (i) having no designated primary replica, and instead, enabling every replica to order the requests that it receives from clients; (ii) using only three communication steps to order requests in the common case; and (iii) involving clients actively in the consensus process. In addition, ezBFT minimizes the potentially negative effect of a byzantine replica on the overall system performance. W…
Byzantine Fault Tolerance (BFT) is one of the most challenging problems in Distributed Machine Learning (DML), defined as the resilience of a fault-tolerant system in the presence of malicious components. Byzantine failures are still difficult to deal with due to their unrestricted nature, which results in the possibility of generating arbitrary data. Significant research efforts are constantly being made to implement BFT in DML. Some recent studies have considered various BFT approaches in DML. However, some aspects are limited, such as the few approaches analyzed, and there is no classificat…
We analyze the impact of transient and Byzantine faults on the construction of a maximal independent set in a general network. We adapt the self-stabilizing algorithm presented by Turau \cite{turau2007linear} for computing such a vertex set. Our algorithm is self-stabilizing and also works under the more difficult context of arbitrary Byzantine faults. Byzantine nodes can prevent nodes close to them from taking part in the independent set for an arbitrarily long time. We give boundaries to their impact by focusing on the set of all nodes excluding nodes at distance 1 or less of Byzantine nodes…
Traditional Byzantine Fault Tolerance (BFT) state machine replication protocols assume a partial synchrony model, leading to a design where a leader replica drives the protocol and is replaced after a timeout. Recently, we witnessed a surge of asynchronous BFT protocols that use randomization to remove the assumptions of bounds on message delivery times, making them more resilient to adverse network conditions. However, these protocols still fall short of being practical across a broad range of scenarios due to their cubic communication costs, use of expensive primitives, and overall protocol …
To implement a blockchain, the trend is now to integrate a non-trivial Byzantine fault tolerant consensus algorithm instead of the seminal idea of waiting to receive blocks to decide upon the longest branch. After a decade of existence, blockchains trade now large amounts of valuable assets and a simple disagreement could lead to disastrous losses. Unfortunately, Byzantine consensus solutions used in blockchains are at best proved correct "by hand" as we are not aware of any of them having been formally verified. In this paper, we propose two contributions: (i) we illustrate the severity of th…
We study a well-known communication abstraction called Byzantine Reliable Broadcast (BRB). This abstraction is central in the design and implementation of fault-tolerant distributed systems, as many fault-tolerant distributed applications require communication with provable guarantees on message deliveries. Our study focuses on fault-tolerant implementations for message-passing systems that are prone to process-failures, such as crashes and malicious behavior. At PODC 1983, Bracha and Toueg, in short, BT, solved the BRB problem. BT has optimal resilience since it can deal with t < n/3 Byzantin…
This paper presents DuoBFT, a Byzantine fault-tolerant protocol that uses trusted components to provide commit decisions in the Hybrid fault model in addition to commit decisions in the BFT model. By doing so, it enables the clients to choose the response fault model for its commands. Internally, DuoBFT commits each client command under both the hybrid and Byzantine models, but since hybrid commits take fewer communication steps and use smaller quorums than BFT commits, clients can benefit from the low-latency commits in the hybrid model. DuoBFT uses a common view-change change protocol to han…
A set of mobile robots is placed at points of an infinite line. The robots are equipped with GPS devices and they may communicate their positions on the line to a central authority. The collection contains an unknown subset of "spies", i.e., byzantine robots, which are indistinguishable from the non-faulty ones. The set of the non-faulty robots need to rendezvous in the shortest possible time in order to perform some task, while the byzantine robots may try to delay their rendezvous for as long as possible. The problem facing a central authority is to determine trajectories for all robots so a…
In this paper, we give a deterministic two-step Byzantine consensus protocol that achieves safety and liveness. A two-step Byzantine consensus protocol only needs two communication steps to commit in the absence of faults. Most two-step Byzantine consensus protocols exploit optimism and require a recovery protocol in the presence of faults. In this paper, we give a simple two-step Byzantine consensus protocol that does not need a recovery protocol.
We define and investigate the consensus problem for a set of $N$ processes embedded on the $d$-dimensional plane, $d\geq 2$, which we call the {\em geoconsensus} problem. The processes have unique coordinates and can communicate with each other through oral messages. In contrast to the literature where processes are individually considered Byzantine, it is considered that all processes covered by a finite-size convex fault area $F$ are Byzantine and there may be one or more processes in a fault area. Similarly as in the literature where correct processes do not know which processes are Byzanti…
In this paper we address Approximate Agreement problem in the Mobile Byzantine faults model. Our contribution is threefold. First, we propose the the first mapping from the existing variants of Mobile Byzantine models to the Mixed-Mode faults model.This mapping further help us to prove the correctness of class MSR (Mean-Subsequence-Reduce) Approximate Agreement algorithms in the Mobile Byzantine fault model, and is of independent interest. Secondly, we prove lower bounds for solving Approximate Agreement under all existing Mobile Byzantine faults models. Interestingly, these lower bounds are d…
In this paper we propose Aleph, a leaderless, fully asynchronous, Byzantine fault tolerant consensus protocol for ordering messages exchanged among processes. It is based on a distributed construction of a partially ordered set and the algorithm for reaching a consensus on its extension to a total order. To achieve the consensus, the processes perform computations based only on a local copy of the data structure, however, they are bound to end with the same results. Our algorithm uses a dual-threshold coin-tossing scheme as a randomization strategy and establishes the agreement in an expected …
The development of fault-tolerant distributed systems that can tolerate Byzantine behavior has traditionally been focused on consensus protocols, which support fully-replicated designs. For the development of more sophisticated high-performance Byzantine distributed systems, more specialized fault-tolerant communication primitives are necessary, however. In this paper, we identify an essential communication primitive and study it in depth. In specifics, we formalize the cluster-sending problem, the problem of sending a message from one Byzantine cluster to another Byzantine cluster in a reliab…
The problem of Byzantine consensus has been key to designing secure distributed systems. However, it is particularly difficult, mainly due to the presence of Byzantine processes that act arbitrarily and the unknown message delays in general networks. Although it is well known that both safety and liveness are at risk as soon as $n/3$ Byzantine processes fail, very few works attempted to characterize precisely the faults that produce safety violations from the faults that produce termination violations. In this paper, we present a new lower bound on the solvability of the consensus problem by d…
One of the most celebrated problems of fault-tolerant distributed computing is the consensus problem. It was shown to abstract a myriad of problems in which processes have to agree on a single value. Consensus applications include fundamental services for the environments of the Cloud or Blockchain. In such challenging environments, malicious behavior is often modeled as adversarial Byzantine faults. At OPODIS 2010, Moste}faoui and Raynal, in short, MR, presented a Byzantine- and intrusion-tolerant solution to consensus in which the decided value cannot be a value proposed only by Byzantine pr…
It is a common belief that Byzantine fault-tolerant solutions for consensus are significantly slower than their crash fault-tolerant counterparts. Indeed, in PBFT, the most widely known Byzantine fault-tolerant consensus protocol, it takes three message delays to decide a value, in contrast with just two in Paxos. This motivates the search for fast Byzantine consensus algorithms that can produce decisions after just two message delays \emph{in the common case}, e.g., under the assumption that the current leader is correct and not suspected by correct processes. The (optimal) two-step latency c…
Minimizing end-to-end latency in geo-replicated systems usually makes it necessary to compromise on resilience, resource efficiency, or throughput performance, because existing approaches either tolerate only crashes, require additional replicas, or rely on a global leader for consensus. In this paper, we eliminate the need for such tradeoffs by presenting Isos, a leaderless replication protocol that tolerates up to $f$ Byzantine faults with a minimum of $3f+1$ replicas. To reduce latency in wide-area environments, Isos relies on an efficient consensus algorithm that allows all participating r…
This paper describes a simple and efficient Binary Byzantine faulty tolerant consensus algorithm using a weak round coordinator and the partial synchrony assumption to ensure liveness. In the algorithm, non-faulty nodes perform an initial broadcast followed by a executing a series of rounds consisting of a single message broadcast until termination. Each message is accompanied by a cryptographic proof of its validity. In odd rounds the binary value 1 can be decided, in even round 0. Up to one third of the nodes can be faulty and termination is ensured within a number of round of a constant fac…
Docker container virtualization technology is being widely adopted in cloud computing environments because of its lightweight and effiency. However, it requires adequate control and management via an orchestrator. As a result, cloud providers are adopting the open-access Kubernetes platform as the standard orchestrator of containerized applications. To ensure applications' availability in Kubernetes, the latter uses Raft protocol's replication mechanism. Despite its simplicity, Raft assumes that machines fail only when shutdown. This failure event is rarely the only reason for a machine's malf…
It has been known since the early 1980s that Byzantine Agreement in the full information, asynchronous model is impossible to solve deterministically against even one crash fault [FLP85], but that it can be solved with probability 1 [Ben83], even against an adversary that controls the scheduling of all messages and corrupts up to $f<n/3$ players [Bra87]. The main downside of [Ben83, Bra87] is that they terminate in $2^{Θ(n)}$ rounds in expectation whenever $f=Θ(n)$. King and Saia [KS16, KS18(arXiv:1812.10169)] developed a polynomial protocol (polynomial rounds, polynomial computation) that is …
Causal consistency is a widely used weak consistency model that allows high availability despite network partitions. There are plenty of research prototypes and industrial deployments of causally consistent distributed systems. However, as far as we know, none of them consider Byzantine faults, except Byz-RCM proposed by Tseng et al. Byz-RCM achieves causal consistency in the client-server model with $3f + 1$ servers where up to $f$ servers may suffer Byzantine faults, but assumes that clients are non-Byzantine. In this work, we present Byz-Gentlerain, the first causal consistency protocol whi…
