With the ever-increasing dataset sizes, several file formats like Parquet, ORC, and Avro have been developed to store data efficiently and to save network and interconnect bandwidth at the price of additional CPU utilization. However, with the advent of networks supporting 25-100 Gb/s and storage devices delivering 1, 000, 000 reqs/sec the CPU has become the bottleneck, trying to keep up feeding data in and out of these fast devices. The result is that data access libraries executed on single clients are often CPU-bound and cannot utilize the scale-out benefits of distributed storage systems. One attractive solution to this problem is to offload data-reducing processing and filtering tasks to the storage layer. However, modifying legacy storage systems to support compute offloading is often tedious and requires extensive understanding of the internals. Previous approaches re-implemented functionality of data processing frameworks and access library for a particular storage system, a duplication of effort that might have to be repeated for different storage systems. In this paper, we introduce a new design paradigm that allows extending programmable object storage systems to embed existing, widely used data processing frameworks and access libraries into the storage layer with minimal modifications. In this approach data processing frameworks and access libraries can evolve independently from storage systems while leveraging the scale-out and availability properties of distributed storage systems. We present one example implementation of our design paradigm using Ceph, Apache Arrow, and Parquet. We provide a brief performance evaluation of our implementation and discuss key results.
BatchWeave: A Consistent Object-Store-Native Data Plane for Large Foundation Model Training
Lionrock AI Lab, China Merchants Group, Hong Kong, China(狮岩人工智能实验室,中国商人集团,香港,中国);Wuhan University, Wuhan, China(武汉大学,武汉,中国);The Chinese University of Hong Kong, Shenzhen, China(香港中文大学(深圳))
Thousand-GPU Large-Scale Training and Optimization Recipe for AI-Native Cloud Embodied Intelligence Infrastructure
AI Infra Team at JDT(JDT人工智能基础设施团队);Tsinghua University(清华大学);Peking University(北京大学);Tianjin University(天津大学);Beihang University(北洋大学);University of Science and Technology of China(中国科学技术大学)
QSimAdv: A Late-Bound, Vendor-Agnostic Architecture for High-Performance Quantum-Circuit Simulation
Pawsey Supercomputing Research Centre(Pawsey超级计算研究中心);The University of Western Australia(西澳大学);Nanjing University of Information Science & Technology(南京信息工程大学);Sina Weibo(新浪微博);Beijing Zhongke Liangshu Technology Co., Ltd(北京中科良数科技有限公司)