DynIMS：一种用于HPC系统内存存储的动态内存控制器
DynIMS: A Dynamic Memory Controller for In-memory Storage on HPC Systems
摘要：
为了提高HPC系统上数据密集型计算的性能，存内计算框架（如Apache Spark和Flink）使用本地DRAM进行数据存储。优化数据存储的内存分配对于为传统HPC计算作业提供性能以及为共享HPC资源的数据密集型应用程序提供吞吐量至关重要。静态配置内存存储的当前实践可能会为计算作业留下不足的空间，或者失去为数据密集型应用程序利用更多可用空间的机会。在本文中，我们将探索动态调整内存存储并为计算作业提供适量空间的技术。我们已经开发了一个动态的内存控制器，DynIMS，它推断的内存需求的计算任务在线，并采用基于反馈的控制模型，以适应内存存储的容量。我们在HPC集群上使用混合HPCC和Spark工作负载测试DynIMS。实验结果表明，DynIMS可以实现高达5倍的性能提高相比，系统与静态内存分配。
英文摘要：
In order to boost the performance of data-intensive computing on HPC systems, in-memory computing frameworks, such as Apache Spark and Flink, use local DRAM for data storage. Optimizing the memory allocation to data storage is critical to delivering performance to traditional HPC compute jobs and throughput to data-intensive applications sharing the HPC resources. Current practices that statically configure in-memory storage may leave inadequate space for compute jobs or lose the opportunity to utilize more available space for data-intensive applications. In this paper, we explore techniques to dynamically adjust in-memory storage and make the right amount of space for compute jobs. We have developed a dynamic memory controller, DynIMS, which infers memory demands of compute tasks online and employs a feedback-based control model to adapt the capacity of in-memory storage. We test DynIMS using mixed HPCC and Spark workloads on a HPC cluster. Experimental results show that DynIMS can achieve up to 5X performance improvement compared to systems with static memory allocations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
