Recent developments in the industry of personal computing led to a greater number of the so-called edge devices. Such devices typically do not collaborate or foresee the possibility of collaboration to offer aggregated storage and computing capabilities. The concept of distributed file system (DFS) is not new to the field of distributed systems, in fact, it is widely used in dedicated infrastructures, for example, in cloud computing applications.
In this work, we discuss reasonable assumptions for an environment composed of edge devices, the main design issues and implementation challenges of a DFS in the given environment and how they would impact this application. Thereafter we define a system model for an environment composed of edge devices while taking into consideration their high mobility and common cases of network partitioning. Next, we describe an architecture for a DFS that withstands the proposed system model while offering most capabilities that a DFS at a dedicated infrastructure would.
We conclude that the development of a distributed file system is a very complex task and, given the broad assumptions of the system model, also hard to verify. Some important aspects of the development lie as future work, but we believe that the developed DFS can be used not only as a tool on it's own, but also as a reference for further development of distributed file systems and, specially, of systems for infrastructures composed of edge devices.
Technische Universität Berlin（柏林工业大学）；University of Glasgow（格拉斯哥大学）；HPI, University of Potsdam（波茨坦大学哈索·普拉特纳研究所）
Distributed file systems are widely used nowadays, yet using their default configurations is often not optimal. At the same time, tuning configuration parameters is typically challenging and time-consuming. It demands expertise and tuning operations can also be expensive. This is especially the case for static parameters, where changes take effect only after a restart of the system or workloads. We propose a novel approach, Magpie, which utilizes deep reinforcement learning to tune static parameters by strategically exploring and exploiting configuration parameter spaces. To boost the tuning o…
The adoption of very low latency persistent memory modules (PMMs) upends the long-established model of disaggregated file system access. Instead, by colocating computation and PMM storage, we can provide applications much higher I/O performance, sub-second application failover, and strong consistency. To demonstrate this, we built the Assise distributed file system, based on a persistent, replicated coherence protocol for managing a set of server-colocated PMMs as a fast, crash-recoverable cache between applications and slower disaggregated storage, such as SSDs. Unlike disaggregated file syst…
Redundant storage maintains the performance of distributed systems under various forms of uncertainty. This paper considers the uncertainty in node access and download service. We consider two access models under two download service models. In one access model, a user can access each node with a fixed probability, and in the other, a user can access a random fixed-size subset of nodes. We consider two download service models. In the first (small file) model, the randomness associated with the file size is negligible. In the second (large file) model, randomness is associated with both the fil…
The XRootD system is used to transfer, store, and cache large datasets from high-energy physics (HEP). In this study we focus on its capability as distributed on-demand storage cache. Through exploring a large set of daily log files between 2020 and 2021, we seek to understand the data access patterns that might inform future cache design. Our study begins with a set of summary statistics regarding file read operations, file lifetimes, and file transfers. We observe that the number of read operations on each file remains nearly constant, while the average size of a read operation grows over ti…
This work examines strategies to handle large shared data objects in distributed storage systems (DSS), while boosting the number of concurrent accesses, maintaining strong consistency guarantees, and ensuring good operation performance. To this respect, we define the notion of fragmented objects:con-current objects composed of a list of fragments (or blocks) that allow operations to manipulate each of their fragments individually. As the fragments belong to the same object, it is not enough that each fragment is linearizable to have useful consistency guarantees in the composed object. Hence,…
The InterPlanetary File System (IPFS) is an hyper-media distribution protocol, addressed by content and identities. It aims to make the web faster, safer, and more open. The JavaScript implementation of IPFS runs on the browser, benefiting from the mass adoption potential that it yields. Startrail takes advantage of the IPFS ecosystem and strives to further evolve it, making it more scalable and performant through the implementation of an adaptive network caching mechanism. Our solution aims to add resilience to IPFS and improve its overall scalability, by avoiding overloading the nodes provid…
Virtualization provides an abstraction layer for the Internet of Things technology to tackle the heterogeneity of the edge networks. It enables the deployment of an application on devices with different architectures to achieve uniformity. This study lays down the fundamentals of a framework for dynamic and event-driven orchestration towards a fully decentralized edge. It provides a blockchain-based delivery platform for containerized applications registered with their resource requirements through a registry on a distributed file system, namely InterPlanetary File System (IPFS). The decentral…
Scripting languages such as Python and R have been widely adopted as tools for the productive development of scientific software because of the power and expressiveness of the languages and available libraries. However, deploying scripted applications on large-scale parallel computer systems such as the IBM Blue Gene/Q or Cray XE6 is a challenge because of issues including operating system limitations, interoperability challenges, parallel filesystem overheads due to the small file system accesses common in scripted approaches, and other issues. We present here a new approach to these problems…
We consider the data shuffling problem in a distributed learning system, in which a master node is connected to a set of worker nodes, via a shared link, in order to communicate a set of files to the worker nodes. The master node has access to a database of files. In every shuffling iteration, each worker node processes a new subset of files, and has excess storage to partially cache the remaining files, assuming the cached files are uncoded. The caches of the worker nodes are updated every iteration, and they should be designed to satisfy any possible unknown permutation of the files in subse…
A peer-to-peer system is a distributed system in which equal nodes (in terms of role and usage) exchange information and services directly. This paper describes a distributed peer-to-peer protocol that allows wifi-enabled smart devices (especially Android smartphones) to exchange data using only wifi. The protocol is designed to allow the automatic establishment of a distributed peer-to-peer network of any size without geography constraint. It is applied to file sharing between devices, but can easily be adapted to support any data sharing. The protocol defines two layers: (1) a kernel layer r…
We consider distributed machine learning at the wireless edge, where a parameter server builds a global model with the help of multiple wireless edge devices that perform computations on local dataset partitions. Edge devices transmit the result of their computations (updates of current global model) to the server using a fixed rate and orthogonal multiple access over an error prone wireless channel. In case of a transmission error, the undelivered packet is retransmitted until successfully decoded at the receiver. Leveraging on the fundamental tradeoff between computation and communication in…
Many science and industry IoT applications necessitate data processing across the edge-to-cloud continuum to meet performance, security, cost, and privacy requirements. However, diverse abstractions and infrastructures for managing resources and tasks across the edge-to-cloud scenario are required. We propose Pilot-Edge as a common abstraction for resource management across the edge-to-cloud continuum. Pilot-Edge is based on the pilot abstraction, which decouples resource and workload management, and provides a Function-as-a-Service (FaaS) interface for application-level tasks. The abstraction…
High-performance object stores are an emerging technology which offers an alternative solution in the field of HPC storage, with potential to address long-standing scalability issues in traditional distributed POSIX file systems due to excessive consistency assurance and metadata prescriptiveness. In this paper we assess the performance of storing object-like data within a standard file system, where the configuration and access mechanisms have not been optimised for object access behaviour, and compare with and investigate the benefits of using an object storage system. Whilst this approach i…
As the capacity of Solid-State Drives (SSDs) is constantly being optimised and boosted with gradually reduced cost, the SSD cluster is now widely deployed as part of the hybrid storage system in various scenarios such as cloud computing and big data processing. However, despite its rapid developments, the performance of the SSD cluster remains largely under-investigated, leaving its sub-optimal applications in reality. To address this issue, in this paper we conduct extensive empirical studies for a comprehensive understanding of the SSD cluster in diverse settings. To this end, we configure a…
Coding theoretic approached have been developed to significantly reduce the communication load in modern distributed computing system. In particular, coded distributed computing (CDC) introduced by Li et al. can efficiently trade computation resources to reduce the communication load in MapReduce like computing systems. For the more general cascaded CDC, Map computations are repeated at r nodes to significantly reduce the communication load among nodes tasked with computing Q Reduce functions s times. In this paper, we propose a novel low-complexity combinatorial design for cascaded CDC which …
Delivering a reproducible environment along with complex and up-to-date software stacks on thousands of distributed and heterogeneous worker nodes is a critical task. The CernVM-File System (CVMFS) has been designed to help various communities to deploy software on worldwide distributed computing infrastructures by decoupling the software from the Operating System. However, the installation of this file system depends on a collaboration with system administrators of the remote resources and an HTTP connectivity to fetch dependencies from external sources. Supercomputers, which offer tremendous…
