arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2204.08400cs.LGcs.PF
收藏

推理时的动态网络适应

Dynamic Network Adaptation at Inference

Stanford University(斯坦福大学)

Daniel Mendoza, Caroline Trippel

摘要:

机器学习(ML)推理是一种实时工作负载,必须符合严格的服务水平目标(SLO),包括延迟和准确性目标。不幸的是,由于固有的模型准确性-延迟权衡、跨应用领域和应用领域内的SLO多样性、SLO随时间的演变、不可预测的查询模式和共址干扰,确保在推理服务系统中不违反SLO是具有挑战性的。在本文中,我们观察到神经网络在推理过程中表现出高度的每输入激活稀疏性。.因此,我们提出了SLO感知神经网络,它根据指定的SLO优化目标和机器利用率,动态地丢弃每个推理查询的节点,从而调整执行的计算量。SLO感知神经网络实现了$1.3-56.7\times$的平均加速,几乎没有精度损失(小于0.3%)。当精度受限时,SLO感知神经网络能够以相同的训练模型以低延迟为一系列精度目标提供服务。当延迟受限时,SLO感知神经网络可以主动缓解由于共址干扰而导致的延迟降级,同时保持高精度以满足延迟约束。

英文摘要:

Machine learning (ML) inference is a real-time workload that must comply with strict Service Level Objectives (SLOs), including latency and accuracy targets. Unfortunately, ensuring that SLOs are not violated in inference-serving systems is challenging due to inherent model accuracy-latency tradeoffs, SLO diversity across and within application domains, evolution of SLOs over time, unpredictable query patterns, and co-location interference. In this paper, we observe that neural networks exhibit high degrees of per-input activation sparsity during inference. . Thus, we propose SLO-Aware Neural Networks which dynamically drop out nodes per-inference query, thereby tuning the amount of computation performed, according to specified SLO optimization targets and machine utilization. SLO-Aware Neural Networks achieve average speedups of $1.3-56.7\times$ with little to no accuracy loss (less than 0.3%). When accuracy constrained, SLO-Aware Neural Networks are able to serve a range of accuracy targets at low latency with the same trained model. When latency constrained, SLO-Aware Neural Networks can proactively alleviate latency degradation from co-location interference while maintaining high accuracy to meet latency constraints.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑