首页/论文/2003.11409

arXiv 2003.114092020-03cs.DChep-ex

Dynamo -- Handling Scientific Data Across Sites and Storage Media

Yutaro Iiyama, Benedikt Maier, Daniel Abercrombie, Maxim Goncharov, Christoph Paus

备注：: 18 pages, 9 figures
发表信息：: Computing and Software for Big Science 5, 11 (2021)
DOI：: 10.1007/s41781-021-00054-2

英文摘要：

Dynamo is a full-stack software solution for scientific data management. Dynamo's architecture is modular, extensible, and customizable, making the software suitable for managing data in a wide range of installation scales, from a few terabytes stored at a single location to hundreds of petabytes distributed across a worldwide computing grid. This article documents the core system design of Dynamo and describes the applications that implement various data management tasks. A brief report is also given on the operational experiences of the system at the CMS experiment at the CERN Large Hadron Collider and at a small scale analysis facility.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Minimizing Content Staleness in Dynamo-Style Replicated Storage Systems

Consistency in data storage systems requires any read operation to return the most recent written version of the content. In replicated storage systems, consistency comes at the price of delay due to large-scale write and read operations. Many applications with low latency requirements tolerate data staleness in order to provide high availability and low operation latency. Using age of information as the staleness metric, we examine a data updating system in which real-time content updates are replicated and stored in a Dynamo-style quorum- based distributed system. A source sends updates to a…

2018-04-02cs.DBcs.DCcs.NI

Complying with Data Handling Requirements in Cloud Storage Systems

In past years, cloud storage systems saw an enormous rise in usage. However, despite their popularity and importance as underlying infrastructure for more complex cloud services, today's cloud storage systems do not account for compliance with regulatory, organizational, or contractual data handling requirements by design. Since legislation increasingly responds to rising data protection and privacy concerns, complying with data handling requirements becomes a crucial property for cloud storage systems. We present PRADA, a practical approach to account for compliance with data handling require…

2020-06-07cs.DCcs.NI

Storage-Optimized Data-Atomic Algorithms for Handling Erasures and Errors in Distributed Storage Systems

Erasure codes are increasingly being studied in the context of implementing atomic memory objects in large scale asynchronous distributed storage systems. When compared with the traditional replication based schemes, erasure codes have the potential of significantly lowering storage and communication costs while simultaneously guaranteeing the desired resiliency levels. In this work, we propose the Storage-Optimized Data-Atomic (SODA) algorithm for implementing atomic memory objects in the multi-writer multi-reader setting. SODA uses Maximum Distance Separable (MDS) codes, and is specifically …

2016-05-05cs.DCcs.ITmath.IT

Consideration for effectively handling parallel workloads on public cloud system

We retrieved and analyzed parallel storage workloads of the FUJITSU K5 cloud service to clarify how to build cost-effective hybrid storage systems. A hybrid storage system consists of fast but low-capacity tier (first tier) and slow but high-capacity tier (second tier). And, it typically consists of either SSDs and HDDs or NVMs and SSDs. As a result, we found that 1) regions for first tier should be assigned only if a workload includes large number of IO accesses for a whole day, 2) the regions that include a large number of IO accesses should be dynamically chosen and moved from second tier t…

2020-08-14cs.DCcs.DScs.OS

D0 Data Handling Operational Experience

We report on the production experience of the D0 experiment at the Fermilab Tevatron, using the SAM data handling system with a variety of computing hardware configurations, batch systems, and mass storage strategies. We have stored more than 300 TB of data in the Fermilab Enstore mass storage system. We deliver data through this system at an average rate of more than 2 TB/day to analysis programs, with a substantial multiplication factor in the consumed data through intelligent cache management. We handle more than 1.7 Million files in this system and provide data delivery to user jobs at Fer…

2003-06-19cs.AIcs.DC

Fragmented ARES: Dynamic Storage for Large Objects

Data availability is one of the most important features in distributed storage systems, made possible by data replication. Nowadays data are generated rapidly and the goal to develop efficient, scalable and reliable storage systems has become one of the major challenges for high performance computing. In this work, we develop a dynamic, robust and strongly consistent distributed storage implementation suitable for handling large objects (such as files). We do so by integrating an Adaptive, Reconfigurable, Atomic Storage framework, called ARES, with a distributed file system, called COBFS, whic…

2022-01-31cs.DC

更多相关论文收起更多论文

Regenerating Codes for Errors and Erasures in Distributed Storage

Regenerating codes are a class of codes proposed for providing reliability of data and efficient repair of failed nodes in distributed storage systems. In this paper, we address the fundamental problem of handling errors and erasures during the data-reconstruction and node-repair operations. We provide explicit regenerating codes that are resilient to errors and erasures, and show that these codes are optimal with respect to storage and bandwidth requirements. As a special case, we also establish the capacity of a class of distributed storage systems in the presence of malicious adversaries. W…

2012-05-23cs.DCcs.ITcs.NI

Design and Evaluation of a Simple Data Interface for Efficient Data Transfer Across Diverse Storage

Modern science and engineering computing environments often feature storage systems of different types, from parallel file systems in high-performance computing centers to object stores operated by cloud providers. To enable easy, reliable, secure, and performant data exchange among these different systems, we propose Connector, a pluggable data access architecture for diverse, distributed storage. By abstracting low-level storage system details, this abstraction permits a managed data transfer service (Globus in our case) to interact with a large and easily extended set of storage systems. Eq…

2020-09-07cs.DC

Handling Confidential Data on the Untrusted Cloud: An Agent-based Approach

Cloud computing allows shared computer and storage facilities to be used by a multitude of clients. While cloud management is centralized, the information resides in the cloud and information sharing can be implemented via off-the-shelf techniques for multiuser databases. Users, however, are very diffident for not having full control over their sensitive data. Untrusted database-as-a-server techniques are neither readily extendable to the cloud environment nor easily understandable by non-technical users. To solve this problem, we present an approach where agents share reserved data in a secur…

2010-12-03cs.CRcs.DCcs.MA

Techniques for Efficiently Handling Power Surges in Fuel Cell Powered Data Centers: Modeling, Analysis, Results

Fuel cells are a promising power source for future data centers, offering high energy efficiency, low greenhouse gas emissions, and high reliability. However, due to mechanical limitations related to fuel delivery, fuel cells are slow to adjust to sudden increases in data center power demands, which can result in temporary power shortfalls. To mitigate the impact of power shortfalls, prior work has proposed to either perform power capping by throttling the servers, or to leverage energy storage devices (ESDs) that can temporarily provide enough power to make up for the shortfall while the fuel…

2018-06-12cs.DC

DSPE：基于纠删码动态空间分区的边云存储系统利润最大化研究

DSPE: Profit Maximization in Edge-Cloud Storage System using Dynamic Space Partitioning with Erasure Code

Indian Institute of Technology Guwahati（印度理工学院古瓦哈提分校）

针对边缘服务器存储容量有限、难应对动态高并发低延迟请求的问题，提出融合纠删码、协作缓存与动态公私分区的利润导向框架，经真实轨迹验证较现有方案提升系统利润5%-8%。

2025-07-30cs.DC

使用 NIO Bench 对机器学习工作负载的存储系统进行基准测试

Benchmarking Storage Systems for Machine Learning Workloads Using NIO Bench

University of California, Santa Cruz（加州大学圣克鲁兹分校）

本文提出NIO Bench框架，通过两层追踪方法刻画六种ML模型的存储访问模式，发现I/O集中于数据准备、模型加载和检查点，且存储访问呈极端幂律，建议优化预取、页缓存固定和突发写入处理。

2026-09-09cs.DCcs.LGcs.PF

Quorum Sensing for Regenerating Codes in Distributed Storage

Distributed storage systems with replication are well known for storing large amount of data. A large number of replication is done in order to provide reliability. This makes the system expensive. Various methods have been proposed over time to reduce the degree of replication and yet provide same level of reliability. One recently suggested scheme is of Regenerating codes, where a file is divided in to parts which are then processed by a coding mechanism and network coding to provide large number of parts. These are stored at various nodes with more than one part at each node. These codes ca…

2013-09-28cs.DCcs.ITmath.IT

TierBase：一种工作负载驱动的成本优化键值存储

TierBase: A Workload-Driven Cost-Optimized Key-Value Store

本文提出面向键值存储的空间-性能成本模型，并据此设计蚂蚁集团的分布式键值存储TierBase，通过缓存与存储层数据同步、预训练压缩、弹性线程和持久内存等技术优化成本，实验及生产案例显示其最高可降低62%成本。

2025-05-10cs.DBcs.DC

利用GPU发起的直接存储访问加速GNN框架中的采样与聚合操作

Accelerating Sampling and Aggregation Operations in GNN Frameworks with GPU Initiated Direct Storage Accesses

UIUC USA（伊利诺伊大学厄巴纳-香槟分校）；NVIDIA USA（英伟达公司）；NVIDIA/UIUC USA（英伟达公司/伊利诺伊大学厄巴纳-香槟分校）

针对大规模图神经网络训练中CPU数据加载瓶颈和内存不足问题，提出GIDS数据加载器，利用GPU直接访问存储、并行容忍延迟并消除缺页异常，使整体训练流程加速高达392倍。

2024-03-06cs.AIcs.ARcs.DC

基于截角八面体量化的误差受限点云压缩

Error-bounded Point Cloud Compression Using Truncated Octahedron Quantization

Temple University（天普大学）；Florida State University（佛罗里达州立大学）；Argonne National Laboratory（阿贡国家实验室）

针对现有点云压缩技术不适用于密集科学数据的问题，提出基于截角八面体量化的XnYZip压缩器，可提升压缩比与处理速度，降低存储开销。

2026-08-04cs.DBcs.DCcs.PF

面向企业级PB级OLAP的数据缓存

Data Caching for Enterprise-Grade Petabyte-Scale OLAP

本文提出Alluxio本地边缘缓存，利用本地SSD缓解PB级OLAP解耦架构下的网络I/O压力，并通过Uber和Meta三年部署验证其处理企业级大规模工作负载的有效性。

2024-06-10cs.DBcs.DC

FluxSieve: 统一流式与分析数据平面以实现可扩展的云可观测性

FluxSieve: Unifying Streaming and Analytical Data Planes for Scalable Cloud Observability

Dynatrace Research（Dynatrace 研究）

FluxSieve通过统一流式与分析数据平面，实现高并发下查询性能的显著提升，同时保持低存储和计算开销。

2026-03-05cs.DBcs.DCcs.PF