In recent years, utilization of heterogeneous hardware other than small core CPU such as GPU, FPGA or many core CPU is increasing. However, when using heterogeneous hardware, barriers of technical skills such as CUDA are high. Based on that, I have proposed environment-adaptive software that enables automatic conversion, configuration, and high performance operation of once written code, according to the hardware to be placed. However, the source language for offloading was mainly C/C++ language applications currently, and there was no research for common offloading for various language applications. In this paper, I study a common method for automatically offloading for various language applications not only in C language but also in Python and Java.
In the recent years, systems using FPGAs, GPUs have increased due to their advantages such as power efficiency compared to CPUs. However, use in systems such as FPGAs and GPUs requires understanding hardware-specific technical specifications such as HDL and CUDA, which is a high hurdle. Based on this background, I previously proposed environment adaptive software that enables automatic conversion, configuration, and high-performance operation of once written code according to the hardware to be placed. As an element of the concept, I proposed a method to automatically offload loop statements o…
In recent years, with the slowing down of Moore's law, utilization of hardware other than CPU such as GPU or FPGA is increasing. However, when using heterogeneous hardware other than CPUs, barriers of technical skills such as CUDA and HDL are high. Based on that, I have proposed environment adaptive software that enables automatic conversion, configuration, and high-performance operation of once written code, according to the hardware to be placed. Partly of the offloading to the GPU and FPGA was automated previously. In this paper, I improve and propose a previous automatic GPU offloading met…
For Open IoT, we have proposed Tacit Computing technology to discover the devices that have data users need on demand and use them dynamically and an automatic GPU offloading technology as an elementary technology of Tacit Computing. However, it can improve limited applications because it only optimizes parallelizable loop statements extraction. Thus, in this paper, to improve performances of more applications automatically, we propose an improved method with reduction of data transfer between CPU and GPU. We evaluate our proposed offloading method by applying it to Darknet and find that it ca…
IoT technologies have been progressed. Now Open IoT concept has attracted attentions which achieve various IoT services by integrating horizontal separated devices and services. For Open IoT era, we have proposed the Tacit Computing technology to discover the devices with necessary data for users on demand and use them dynamically. However, existing Tacit Computing does not care about performance and operation cost. Therefore in this paper, we propose an automatic GPU offloading technology as an elementary technology of Tacit Computing which uses Genetic Algorithm to extract appropriate offloa…
Evaluating OpenMP Offloading for Intra-node Multi-GPU Programming across NVIDIA, AMD, and Intel Architectures: A 3D Heat Transfer Case Study
Rudolfovo-Science and Technology Centre Novo mesto（鲁多尔沃夫诺沃梅斯托科学与技术中心）；Faculty of Mechanical Engineering, University of Ljubljana（卢布尔雅那大学机械工程学院）；Department of Computer Science, FSTM, University of Luxembourg（卢森堡大学科学技术学院计算机科学系）
In recent years, utilization of heterogeneous hardware other than small core CPU such as GPU, FPGA or many core CPU is increasing. However, when using heterogeneous hardware, barriers of technical skills such as OpenMP, CUDA and OpenCL are high. Based on that, I have proposed environment-adaptive software that enables automatic conversion, configuration, and high performance operation of once written code, according to the hardware to be placed. However, including existing technologies, there has been no research to properly and automatically offload the mixed offloading destination environmen…
In recent years, utilization of heterogeneous hardware other than small core CPU such as GPU, FPGA or many core CPU is increasing. However, when using heterogeneous hardware, barriers of technical skills such as CUDA are high. Based on that, I have proposed environment-adaptive software that enables automatic conversion, configuration, and high performance and low power operation of once written code, according to the hardware to be placed. I also have verified performance improvement of automatic GPU and FPGA offloading so far. In this paper, I verify low power operation with environment adap…
When using heterogeneous hardware other than CPUs, barriers of technical skills such as OpenCL are high. Based on that, I have proposed environment adaptive software that enables automatic conversion, configuration, and high-performance operation of once written code, according to the hardware to be placed. Partly of the offloading to the GPU was automated previously. In this paper, I propose and evaluate an automatic extraction method of appropriate offload target loop statements of source code as the first step of offloading to FPGA. I evaluate the effectiveness of the proposed method in mul…
In recent years, with the prediction of Moore's law slowing down, utilization of hardware other than CPU such as FPGA which is energy effective is increasing. However, when using heterogeneous hardware other than CPUs, barriers of technical skills such as OpenCL are high. Based on that, I have proposed environment adaptive software that enables automatic conversion, configuration, and high-performance operation of once written code, according to the hardware to be placed. Partly of the offloading to the GPU was automated previously. In this paper, I propose and evaluate an automatic extraction…
In recent years, with the prediction of Moore's law slowing down, utilization of hardware other than CPU such as FPGA which is energy effective is increasing. However, when using heterogeneous hardware other than CPUs, barriers of technical skills such as OpenCL are high. Based on that, I have proposed environment adaptive software that enables automatic conversion, configuration, and high-performance operation of once written code, according to the hardware to be placed. Partly of the offloading to the GPU was automated previously. In this paper, I propose and evaluate an automatic extraction…
OpenMC is an open source Monte Carlo neutral particle transport application that has recently been ported to GPU using the OpenMP target offloading model. We examine the performance of OpenMC at scale on the Frontier, Polaris, and Aurora supercomputers, demonstrating that performance portability has been achieved by OpenMC across all three major GPU vendors (AMD, NVIDIA, and Intel). OpenMC's GPU performance is compared to both the traditional CPU-based version of OpenMC as well as several other state-of-the-art CPU-based Monte Carlo particle transport applications. We also provide historical c…
ZenFlow: Enabling Stall-Free Offloading for LLM Training
University of Virginia（弗吉尼亚大学）；University of California, Merced（加州大学梅尔塞德斯分校）；Argonne National Laboratory（阿贡国家实验室）；Snowflake AI Research（Snowflake人工智能研究）
In recent years, utilization of heterogeneous hardware other than small core CPU such as GPU, FPGA or many core CPU is increasing. However, when using heterogeneous hardware, barriers of technical skills such as OpenMP, CUDA and OpenCL are high. Based on that, I have proposed environment-adaptive software that enables automatic conversion, configuration, and high performance operation of once written code, according to the hardware to be placed. However, although the conversion of the code according to the migration destination environment has been studied so far, there has been no research to…
TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading
SKLP, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China（SKLP，计算技术研究所，中国科学院，北京）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京）
DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs
Institute of Automation, CAS（中国科学院自动化研究所）；School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）；Shanghai Jiao Tong University（上海交通大学）；AiRiA
Deep neural networks (DNNs) have emerged as successful solutions for variety of artificial intelligence applications, but their very large and deep models impose high computational requirements during training. Multi-GPU parallelization is a popular option to accelerate demanding computations in DNN training, but most state-of-the-art multi-GPU deep learning frameworks not only require users to have an in-depth understanding of the implementation of the frameworks themselves, but also apply parallelization in a straight-forward way without optimizing GPU utilization. In this work, we propose a…
Benchmarking and comparing performance of a scientific simulation across hardware platforms is a complex task. When the simulation in question is constructed with an asynchronous, many-task (AMT) runtime offloading work to GPUs, the task becomes even more complex. In this paper, we discuss the use of a uniquely suited performance measurement library, APEX, to capture the performance behavior of a simulation built on HPX, a highly scalable, distributed AMT runtime. We examine the performance of the astrophysics simulation carried-out by Octo-Tiger on two different supercomputing architectures. …
EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?
The Hong Kong University of Science and Technology（香港科技大学）；Huawei Technologies Ltd.（华为技术有限公司）；The University of Hong Kong（香港大学）；Sun Yat-sen University（中山大学）
Linear Programs (LPs) appear in a large number of applications and offloading them to the GPU is viable to gain performance. Existing work on offloading and solving an LP on GPU suggests that performance is gained from large sized LPs (typically 500 constraints, 500 variables and above). In order to gain performance from GPU for applications involving small to medium sized LPs, we propose batched solving of a large number of LPs in parallel. In this paper, we present the design and CUDA implementation of our batched LP solver library, keeping memory coalescent access, reduced CPU-GPU memory tr…
