JIZHI:一个用于百度Web规模在线推理的快速且经济高效的模型即服务系统
JIZHI: A Fast and Cost-Effective Model-As-A-Service System for Web-Scale Online Inference at Baidu
摘要:
在现代互联网产业中,基于深度学习的推荐系统已经成为诸如搜索引擎、新闻推送和短视频剪辑等广泛应用的不可或缺的构建块。然而,对于来自数十亿用户的随时间变化的网络规模流量,以具有成本效益的方式携带经过良好训练的深度模型用于在线实时推理服务仍然具有挑战性。在这项工作中,我们提出了JIZHI -一个模型即服务系统-每秒处理数亿个在线推理请求到具有超过万亿个稀疏参数的巨大深度模型,用于百度公司的二十多个实时推荐服务。在JIZHI中,每个推荐请求的推理工作流被转换为分级事件驱动的流水线(SEDP),其中流水线中的每个节点引用分级计算或I/O密集型任务处理器。随着实时推理请求流量的到来,每个模块化处理器可以以完全并行的方式运行并单独管理。此外,集智引入异构和分层存储,通过减少不必要的计算和超稀疏模型参数引起的潜在数据访问延迟,进一步加速在线推理过程。此外,一个智能资源管理器已被部署,以最大限度地提高吞吐量的JIZHI共享基础设施,通过搜索最佳的资源分配计划,从历史日志和微调的负载削减政策的中间系统反馈。通过大量的实验,从端到端的服务延迟、系统吞吐量和资源消耗等方面证明了JIZHI的优势。JIZHI帮助百度节省了超过一千万美元的硬件和公用事业成本,同时在不牺牲推理效率的情况下处理了200%的流量。
英文摘要:
In modern internet industries, deep learning based recommender systems have became an indispensable building block for a wide spectrum of applications, such as search engine, news feed, and short video clips. However, it remains challenging to carry the well-trained deep models for online real-time inference serving, with respect to the time-varying web-scale traffics from billions of users, in a cost-effective manner. In this work, we present JIZHI - a Model-as-a-Service system - that per second handles hundreds of millions of online inference requests to huge deep models with more than trillions of sparse parameters, for over twenty real-time recommendation services at Baidu, Inc. In JIZHI, the inference workflow of every recommendation request is transformed to a Staged Event-Driven Pipeline (SEDP), where each node in the pipeline refers to a staged computation or I/O intensive task processor. With traffics of real-time inference requests arrived, each modularized processor can be run in a fully asynchronized way and managed separately. Besides, JIZHI introduces heterogeneous and hierarchical storage to further accelerate the online inference process by reducing unnecessary computations and potential data access latency induced by ultra-sparse model parameters. Moreover, an intelligent resource manager has been deployed to maximize the throughput of JIZHI over the shared infrastructure by searching the optimal resource allocation plan from historical logs and fine-tuning the load shedding policies over intermediate system feedback. Extensive experiments have been done to demonstrate the advantages of JIZHI from the perspectives of end-to-end service latency, system-wide throughput, and resource consumption. JIZHI has helped Baidu saved more than ten million US dollars in hardware and utility costs while handling 200% more traffics without sacrificing inference efficiency.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。