Decentralized data storage systems like the Interplanetary Filesystem (IPFS) are becoming increasingly popular, e. g., as a data layer in blockchain applications and for sharing content in a censorship-resistant manner. In IPFS, data is hosted by an open set of nodes and data requests are broadcast to connected peers in addition to being routed via a distributed hash table (DHT). In this paper, we present a passive monitoring methodology that exploits this design for obtaining data requests from a significant and upscalable portion of nodes. Using an implementation of our approach for the IPFS network and data collected over a period of fifteen months, we demonstrate how our methodology enables profound insights into, among other things: the size of the IPFS network, activity levels and structure, and content popularity distributions. We furthermore present that our methodology can be abused for attacks on users' privacy. For example, we were able to identify and successfully surveil the IPFS nodes corresponding to public IPFS/HTTP gateways.We give a detailed analysis of the mechanics and reasons behind implied privacy threats and discuss possible countermeasures.
Recent years have witnessed growing consolidation of web operations. For example, the majority of web traffic now originates from a few organizations, and even micro-websites often choose to host on large pre-existing cloud infrastructures. In response to this, the "Decentralized Web" attempts to distribute ownership and operation of web services more evenly. This paper describes the design and implementation of the largest and most widely used Decentralized Web platform - the InterPlanetary File System (IPFS) - an open-source, content-addressable peer-to-peer network that provides distributed…
AI requires heavy amounts of storage and compute. As a result, AI developers are regular users of centralised cloud services such as AWS, GCP and Azure, compute environments such as Jupyter and Colab notebooks, and AI Hubs such as HuggingFace and ActiveLoop. There services are associated with certain benefits and limitations that stem from the underlying infrastructure and governance systems with which they are built. These limitations include high costs, lack of monetization and reward, lack of control and difficulty of reproducibility. At the same time, there are few libraries that allow dat…
The InterPlanetary File System~(IPFS) is a popular decentralized peer-to-peer network for exchanging data. While there are many use cases for IPFS, the success of these use cases depends on the network. In this paper, we provide a passive measurement study of the IPFS network, investigating peer dynamics and curiosities of the network. With the help of our measurement, we estimate the network size and confirm the results of previous active measurement studies.
In IPFS content identifiers are constructed based on the item's data therefore the binding between an item's identifier and its data can be deterministically verified. Nevertheless, once an item is modified, its identifier also changes. Therefore when it comes to mutable content there is a need for keeping track of the "latest" IPFS identifier. This is achieved using naming protocols on top of IPFS, such as IPNS and DNSlink, that map a constant name to an IPFS identifier, allowing at the same time content owners to update these mappings. Nevertheless, IPNS relies on a cryptographic key pair th…
The InterPlanetary File System (IPFS) is a novel decentralised storage architecture, which attempts to provide decentralised cloud storage by building on founding principles of P2P networking and content addressing. IPFS is used by more than 230k peers per week and serves tens of millions of requests per day, which makes it an interesting large-scale operational network to study. While it is used as a building block in several projects and studies, its inner workings, properties, and implications have only been marginally explored in research. Thus, we provide an overview of the IPFS design an…
The InterPlanetary File System (IPFS) is a hypermedia distribution protocol enabling the creation of completely distributed applications. One of the most efficient and effective ways to distribute information is through notifications, with a producer of content (publisher) sharing content with other interested parts (subscribers). IPFS already implements topic-based publish-subscribe systems under an experimental flag. The goal of this work is to advance on that, by developing a content-based pub-sub system (with subscriptions as predicates about event content) to disseminate information on to…
Decentralized, distributed storage offers a way to reduce the impact of data silos as often fostered by centralized cloud storage. While the intentions of this trend are not new, the topic gained traction due to technological advancements, most notably blockchain networks. As a consequence, we observe that a new generation of peer-to-peer data networks emerges. In this survey paper, we therefore provide a technical overview of the next generation data networks. We use select data networks to introduce general concepts and to emphasize new developments. Specifically, we provide a deeper outline…
The InterPlanetary File System (IPFS) is an hyper-media distribution protocol, addressed by content and identities. It aims to make the web faster, safer, and more open. The JavaScript implementation of IPFS runs on the browser, benefiting from the mass adoption potential that it yields. Startrail takes advantage of the IPFS ecosystem and strives to further evolve it, making it more scalable and performant through the implementation of an adaptive network caching mechanism. Our solution aims to add resilience to IPFS and improve its overall scalability, by avoiding overloading the nodes provid…
The proliferation of resourceful mobile devices that store rich, multidimensional and privacy-sensitive user data motivate the design of federated learning (FL), a machine-learning (ML) paradigm that enables mobile devices to produce an ML model without sharing their data. However, the majority of the existing FL frameworks rely on centralized entities. In this work, we introduce IPLS, a fully decentralized federated learning framework that is partially based on the interplanetary file system (IPFS). By using IPLS and connecting into the corresponding private IPFS network, any party can initia…
This paper presents an architecture, based on Distributed Ledger Technologies (DLTs) and Decentralized File Storage (DFS) systems, to support the use of Personal Information Management Systems (PIMS). DLT and DFS are used to manage data sensed by mobile users equipped with devices with sensing capability. DLTs guarantee the immutability, traceability and verifiability of references to personal data, that are stored in DFS. In fact, the inclusion of data digests in the DLT makes it possible to obtain an unalterable reference and a tamper-proof log, while remaining compliant with the regulations…
The design of caching algorithms to maximize hit probability has been extensively studied. In this paper, we associate each content with a utility, which is a function of either the corresponding content hit rate or hit probability. We formulate a cache optimization problem to maximize the sum of utilities over all contents under stationary and ergodic request processes. This problem is non-convex in general but we reformulate it as a convex optimization problem when the inter-request time (irt) distribution has a non-increasing hazard rate function. We provide explicit optimal solutions for s…
The blockchain technology has been changing our daily lives since Bitcoin - i.e., the first decentralized cryptocurrency - was invented and released as open-source software by an unidentified person or a group called Satoshi Nakamoto in 2009. Of many applications which can be implemented based on the blockchain, storage is an important one, a notable example of which is the InterPlanetary File System (IPFS). IPFS is a distributed web based on a peer-to-peer hypermedia protocol to make the web faster, safer, and more open and focuses on public accessible files. To provide a solution for private…
Artificial Intelligence Research Institute of the Spanish National Research Council (IIIA-CSIC)（西班牙国家研究委员会人工智能研究所（IIIA-CSIC））；University of Southern California（南加州大学）
We present the Limited-range Online Routing Problem (LORP), which involves a team of Unmanned Aerial Vehicles (UAVs) with limited communication range that must autonomously coordinate to service task requests. We first show a general approach to cast this dynamic problem as a sequence of decentralized task allocation problems. Then we present two solutions both based on modeling the allocation task as a Markov Random Field to subsequently assess decisions by means of the decentralized Max-Sum algorithm. Our first solution assumes independence between requests, whereas our second solution also …
Plenty of research on smart mobility is currently devoted to the inclusion of novel decentralized software architectures to these systems, due to the inherent advantages in terms of transparency, traceability, trustworthiness. MOVO is a decentralized application (dApp) for smart mobility. It includes: (i) a module for collecting data from vehicles and smartphones sensors; (ii) a component for interacting with Distributed Ledger Technologies (DLT) and Decentralized File Storages (DFS), for storing and validating sensor data; (iii) a module for "offline" interaction between devices. The dApp con…
In-network caching is recognized as an effective solution to offload content servers and the network. A cache service provider (SP) always has incentives to better utilize its cache resources by taking into account diverse roles that content providers (CPs) play, e.g., their business models, traffic characteristics, preferences. In this paper, we study the cache resource allocation problem in a Multi-Cache Multi-CP environment. We propose a cache partitioning approach, where each cache can be partitioned into slices with each slice dedicated to a content provider. We propose a content-obliviou…
With the continuous expansion of Internet of Things (IoT) devices, edge computing mode has emerged in recent years to overcome the shortcomings of traditional cloud computing mode, such as high delay, network congestion, and large resource consumption. Thus, edge-thing systems will replace the classic cloud-thing/cloud-edge-thing systems and become mainstream gradually, where IoT devices can offload their tasks to neighboring edge nodes. A common problem is how to utilize edge computing resources. For the sake of fairness, double auction can be used in the edge-thing system to achieve an effec…
Serverless computing is becoming widely adopted among cloud providers, thus making increasingly popular the Function-as-a-Service (FaaS) programming model, where the developers realize services by packaging sequences of stateless function calls. The current technologies are very well suited to data centers, but cannot provide equally good performance in decentralized environments, such as edge computing systems, which are expected to be typical for Internet of Things (IoT) applications. In this paper, we fill this gap by proposing a framework for efficient dispatching of stateless tasks to in-…
Due to the drastic increase of mobile traffic, wireless caching is proposed to serve repeated requests for content download. To determine the caching scheme for decentralized caching networks, the content preference learning problem based on mobility prediction is studied. We first formulate preference prediction as a decentralized regularized multi-task learning (DRMTL) problem without considering the mobility of mobile terminals (MTs). The problem is solved by a hybrid Jacobian and Gauss-Seidel proximal multi-block alternating direction method (ADMM) based algorithm, which is proven to condi…
Software-Defined Network (SDN) is a new arising terminology of network architecture with outstanding features of orchestration by decoupling the control plane and the data plane in each network element. Even though it brings several benefits, SDN is vulnerable to a diversity of attacks. Abusing the single point of failure in the SDN controller component, hackers can shut down all network operations. More specifics, a malicious OpenFlow application can access to SDN controller to carry out harmful actions without any limitation owing to the lack of the access control mechanism as a standard in …
Edge computing servers like cloudlets from different service providers compensate scarce computational, memory, and energy resources of mobile devices, are distributed across access networks. However, depending on the mobility pattern and dynamically varying computational requirements of associated mobile devices, cloudlets at different parts of the network become either overloaded or under-loaded. Hence, load balancing among neighboring cloudlets appears to be an essential research problem. Nonetheless, the existing load balancing frameworks are unsuitable for low-latency applications. Thus, …
The dominance of a few big companies in the storage market arising various concerns including single point of failure, privacy violation, and oligopoly. To eliminate the dependency on such a centralized storage architecture, several Decentralized Storage Network (DSN) schemes such as Filecoin, Sia, and Storj have been introduced. DSNs leverage blockchain technology to create a storage platform such that the micro storage providers can also participate in the storage market. To verify the accurate data storage by the storage providers during a storage contract, DSNs apply a Proof of Storage (Po…
