arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2209.08625cs.LGcs.PF
收藏

使用自动层缓存提高基于DNN的软件服务的性能

Improving the Performance of DNN-based Software Services using Automated Layer Caching

University of Calgary(卡尔加里大学);University of South Carolina(南卡罗来纳大学)

Mohammadamin Abedi, Yanni Iouannou, Pooyan Jamshidi, Hadi Hemmati

摘要:

深度神经网络(DNN)已经成为包括基于Web的服务在内的许多应用领域中的重要组件。这些服务中的许多都需要高吞吐量和(接近)实时特性,例如,对用户的请求作出响应或反应,或及时处理传入的数据流。然而,DNN设计的趋势是使用具有许多层和参数的更大模型,以获得更准确的结果。尽管这些模型通常是预先训练的,但这种大型模型中的计算复杂性仍然相对显著,阻碍了低推理延迟。实现高速缓存机制是用于加快服务响应时间的典型系统工程解决方案。然而,传统的缓存通常不适合基于DNN的服务。在本文中,我们提出了一种端到端的自动化解决方案,以提高基于DNN的服务在计算复杂性和推理延迟方面的性能。我们的缓存方法采用了DNN模型自升华和早期退出的思想。所提出的解决方案是一种自动在线层缓存机制,如果早期退出中的一个缓存模型对最终预测足够自信,则允许在推理时间期间提前退出大型模型。本文的主要贡献之一是,我们已经实现了在线缓存的想法,这意味着缓存模型不需要访问训练数据,并在运行时仅根据传入的数据执行,使其适用于使用预训练模型的应用程序。我们对两个下游任务(人脸和对象分类)的实验结果表明,平均而言,缓存可以降低这些服务的计算复杂性高达58\%(在FLOPs计数方面),并将其推理延迟提高到46\%,准确性降低到零。

英文摘要:

Deep Neural Networks (DNNs) have become an essential component in many application domains including web-based services. A variety of these services require high throughput and (close to) real-time features, for instance, to respond or react to users' requests or to process a stream of incoming data on time. However, the trend in DNN design is toward larger models with many layers and parameters to achieve more accurate results. Although these models are often pre-trained, the computational complexity in such large models can still be relatively significant, hindering low inference latency. Implementing a caching mechanism is a typical systems engineering solution for speeding up a service response time. However, traditional caching is often not suitable for DNN-based services. In this paper, we propose an end-to-end automated solution to improve the performance of DNN-based services in terms of their computational complexity and inference latency. Our caching method adopts the ideas of self-distillation of DNN models and early exits. The proposed solution is an automated online layer caching mechanism that allows early exiting of a large model during inference time if the cache model in one of the early exits is confident enough for final prediction. One of the main contributions of this paper is that we have implemented the idea as an online caching, meaning that the cache models do not need access to training data and perform solely based on the incoming data at run-time, making it suitable for applications using pre-trained models. Our experiments results on two downstream tasks (face and object classification) show that, on average, caching can reduce the computational complexity of those services up to 58\% (in terms of FLOPs count) and improve their inference latency up to 46\% with low to zero reduction in accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑