One Size Does Not Fit All:Quantifying and Exposing the Accuracy-Latency Trade-off in机器学习云服务API via Tolerance Tiers
One Size Does Not Fit All: Quantifying and Exposing the Accuracy-Latency Trade-off in Machine Learning Cloud Service APIs via Tolerance Tiers
摘要:
当今的云服务架构遵循“一刀切”的部署策略,其中向最终用户提供相同的服务版本实例化。然而,消费者是广泛的,不同的应用程序有不同的准确性和响应能力的要求,这正如我们所证明的,使“一刀切”的方法在实践中效率低下。我们使用一个生产级的语音识别引擎,它为数千名用户提供服务,以及一个基于开源计算机视觉的系统,来解释我们的观点。为了克服“一刀切”方法的局限性,我们建议使用Tolerance Tiers,其中每个MLaaS层都公开一个准确性/响应特性,消费者可以通过编程方式选择一个层。我们评估了我们关于基于CPU的自动语音识别(ASR)引擎和部署在CPU和GPU上的用于图像分类的尖端神经网络的提案。结果表明,我们提出的方法提供了一个MLaaS云服务架构,可以由最终API用户或消费者调整,以优于传统的“一刀切”的方法。
英文摘要:
Today's cloud service architectures follow a "one size fits all" deployment strategy where the same service version instantiation is provided to the end users. However, consumers are broad and different applications have different accuracy and responsiveness requirements, which as we demonstrate renders the "one size fits all" approach inefficient in practice. We use a production-grade speech recognition engine, which serves several thousands of users, and an open source computer vision based system, to explain our point. To overcome the limitations of the "one size fits all" approach, we recommend Tolerance Tiers where each MLaaS tier exposes an accuracy/responsiveness characteristic, and consumers can programmatically select a tier. We evaluate our proposal on the CPU-based automatic speech recognition (ASR) engine and cutting-edge neural networks for image classification deployed on both CPUs and GPUs. The results show that our proposed approach provides an MLaaS cloud service architecture that can be tuned by the end API user or consumer to outperform the conventional "one size fits all" approach.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。