随着智能设备和物联网的无处不在的部署,用于机器学习推理的数据源越来越多地移动到网络边缘。现有的机器学习推理平台通常假设同构基础设施,并且不考虑更复杂和分层的计算基础设施,包括边缘设备、本地集线器、边缘计算中心和云计算中心。另一方面,最近的AutoML努力为异构环境的模型压缩、修剪和量化提供了可行的解决方案;对于机器学习模型,现在我们可以很容易地找到甚至生成一系列在准确性和效率之间具有不同权衡的模型。我们设计并实现了JellyBean,一个在异构基础设施上服务和优化机器学习推理工作流的系统。给定服务级别目标(例如,吞吐量、准确性),JellyBean选择符合准确性目标的最具成本效益的模型,并决定如何在不同的基础架构层中部署它们。评估表明,与最先进的模型选择和工人分配解决方案相比,JellyBean将视觉问答的总服务成本降低了58%,将NVIDIA AI城市挑战赛的车辆跟踪成本降低了36%。JellyBean还优于先前的ML服务系统(例如,Spark on the Cloud),服务成本高达5倍。
英文摘要:
With the advent of ubiquitous deployment of smart devices and the Internet of Things, data sources for machine learning inference have increasingly moved to the edge of the network. Existing machine learning inference platforms typically assume a homogeneous infrastructure and do not take into account the more complex and tiered computing infrastructure that includes edge devices, local hubs, edge datacenters, and cloud datacenters. On the other hand, recent AutoML efforts have provided viable solutions for model compression, pruning and quantization for heterogeneous environments; for a machine learning model, now we may easily find or even generate a series of models with different tradeoffs between accuracy and efficiency. We design and implement JellyBean, a system for serving and optimizing machine learning inference workflows on heterogeneous infrastructures. Given service-level objectives (e.g., throughput, accuracy), JellyBean picks the most cost-efficient models that meet the accuracy target and decides how to deploy them across different tiers of infrastructures. Evaluations show that JellyBean reduces the total serving cost of visual question answering by up to 58%, and vehicle tracking from the NVIDIA AI City Challenge by up to 36% compared with state-of-the-art model selection and worker assignment solutions. JellyBean also outperforms prior ML serving systems (e.g., Spark on the cloud) up to 5x in serving costs.
GraphFlow: A Graph-Based Workflow Management for Efficient LLM-Agent Serving
School of Cyber Science and Engineering, Xi'an Jiaotong University, Xi'an, China(西安交通大学计算机科学与工程学院);School of Artificial Intelligence, Shandong University, Jinan, China(山东大学人工智能学院);Shanghai Advanced Research Institute, Chinese Academy of Sciences, Shanghai, China(中国科学院上海先进研究院)
Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs
University of North Carolina, Chapel Hill(北卡罗来纳大学教堂山分校);Microsoft(微软);Carnegie Mellon University(卡内基梅隆大学);Amazon(亚马逊);University of California, Santa Barbara(加州大学圣芭芭拉分校)
Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
School of Computing and Information Systems, The University of Melbourne(墨尔本大学计算与信息系统学院);School of Computer Science and Technology, Huazhong University of Science and Technology(华中科技大学计算机科学与技术学院)
Scientific Computing, Rutherford Appleton Laboratory Science and Technology Facilities Council (STFC)(科学计算、拉瑟福德-苹果顿实验室科学与技术设施委员会);Department of Engineering Science, University of Oxford(工程科学系,牛津大学)
本文针对 ML 工作流中大量中间数据的管理问题,设计并实现了一个数据虚拟化服务,重点介绍其服务架构与运营,并已支持六个 ML 应用及多个工作流,具备良好的可扩展性。
From Data to Decision: Data-Centric Infrastructure for Reproducible ML in Collaborative eScience
Dept. of Industrial and Systems Engineering(工业与系统工程系);University of Southern California(南加州大学);Information Sciences Institute(信息科学研究所);Dept. of Computer Science(计算机科学系);Dept. of Ophthalmology(眼科学系)