OTAS:一种基于 Token 适配的弹性 Transformer 服务系统
OTAS: An Elastic Transformer Serving System via Token Adaptation
AI总结:
本文提出首个为 Transformer 模型定制的弹性服务系统 OTAS,通过 token 适配(添加提示 token 提升精度、移除冗余 token 加速推理)、应用感知选择性批处理及在线优化,有效应对动态负载与异构需求,将系统效用提升至少 18.2%。
摘要:
Transformer 模型驱动的架构已成为云服务的支柱,并持续重塑我们的社会。然而,动态的查询负载和异构的用户需求严重挑战了当前的 Transformer 服务系统,这些系统依赖预训练基础模型的多个变体(即不同大小)来适应不同的服务需求。不幸的是,由于额外的训练成本和过高的 I/O 延迟,这种机制不适用于大型 Transformer 模型。在本文中,我们介绍了 OTAS,这是首个通过探索轻量级 token 管理专门为 Transformer 模型定制的弹性服务系统。我们开发了一种名为 token adaptation(token 适配)的新思路,通过添加提示 token 来提高准确性,并移除冗余 token 来加速推理。为了应对波动的查询负载和多样的用户请求,我们通过应用感知的选择性批处理和在线 token 适配来增强 OTAS。OTAS 首先将具有相似服务级别目标的传入查询进行批处理,以提高入口吞吐量。然后,为了在 token 增加的开销和准确性提升的潜力之间取得权衡,OTAS 通过解决一个优化问题来自适应地调整 token 执行策略。我们使用多个数据集实现并评估了 OTAS 的原型,结果表明 OTAS 将系统效用提高了至少 18.2%。
英文摘要:
Transformer model empowered architectures have become a pillar of cloud services that keeps reshaping our society. However, the dynamic query loads and heterogeneous user requirements severely challenge current transformer serving systems, which rely on pre-training multiple variants of a foundation model, i.e., with different sizes, to accommodate varying service demands. Unfortunately, such a mechanism is unsuitable for large transformer models due to the additional training costs and excessive I/O delay. In this paper, we introduce OTAS, the first elastic serving system specially tailored for transformer models by exploring lightweight token management. We develop a novel idea called token adaptation that adds prompting tokens to improve accuracy and removes redundant tokens to accelerate inference. To cope with fluctuating query loads and diverse user requests, we enhance OTAS with application-aware selective batching and online token adaptation. OTAS first batches incoming queries with similar service-level objectives to improve the ingress throughput. Then, to strike a tradeoff between the overhead of token increment and the potentials for accuracy improvement, OTAS adaptively adjusts the token execution strategy by solving an optimization problem. We implement and evaluate a prototype of OTAS with multiple datasets, which show that OTAS improves the system utility by at least 18.2%.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。