arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.08470cs.LGcs.AI
收藏

上下文感知助手选择:提升大型语言模型推理加速

Context-Aware Assistant Selection for Improved Inference Acceleration with Large Language Models

Chandar Research Lab(钱达尔研究实验室);Mila(米拉研究所);Université de Montréal(蒙特利尔大学);Polytechnique Montréal(蒙特利尔理工学院)

Jerry Huang, Prasanna Parthasarathi, Mehdi Rezagholizadeh, Sarath Chandar

AI总结:

本研究将辅助解码中的草稿模型选择建模为上下文多臂老虎机,通过离线数据训练对齐策略,无需先验知识即可在多领域加速LLM推理,并验证了其灵活性与有效性。

摘要:

尽管大型语言模型(LLMs)被广泛采用,但在资源受限条件下使用它们仍然困难重重,且其不断增长的规模进一步提高了使用门槛。一个显著问题是自回归生成带来的高延迟,这使得大型LLMs的使用依赖于先进的计算基础设施。辅助解码(assisted decoding)通过让较小的草稿模型引导较大的目标模型生成,缓解了这一问题,但其效果仍依赖于两个模型之间的对齐程度。因此,如果草稿模型在某个领域相对于目标模型能力不足,性能可能会下降。另一种方法是利用多个草稿模型来更好地覆盖目标模型的专长领域,但当有多个黑盒草稿模型可用时,在没有关于其构建细节的情况下选择助手可能很困难。为了更好地理解这一决策问题,我们将其视为一个上下文多臂老虎机(contextual bandit)问题,其中策略必须根据上下文选择草稿模型。我们证明,即使对草稿模型没有先验知识,仅从独立的草稿/目标模型的输出中创建离线数据集,并训练一个基于这些输出对齐的策略,只要候选模型有效,就能在多个领域加速性能。进一步的结果表明,在具有多个辅助解码候选的各种设置中,这一方法均有效,突显了其灵活性和此类决策在加速中的优势作用。

英文摘要:

Despite their widespread adoption, large language models (LLMs) remain prohibitive to use under resource constraints, with their ever growing sizes only increasing the barrier for use. One noted issue is the high latency associated with auto-regressive generation, rendering large LLMs use dependent on advanced computing infrastructure. Assisted decoding, where a smaller draft model guides a larger target model's generation, has helped alleviate this, but remains dependent on alignment between the two models. Thus if the draft model is insufficiently capable on some domain relative to the target model, performance can degrade. Alternatively, one can leverage multiple draft models to better cover the expertise of the target, but when multiple black-box draft models are available, selecting an assistant without details about its construction can be difficult. To better understand this decision making problem, we observe it as a contextual bandit, where a policy must choose a draft model based on a context. We show that even without prior knowledge of the draft models, creating an offline dataset from only outputs of independent draft/target models and training a policy over the alignment of these outputs can accelerate performance on multiple domains provided the candidates are effective. Further results show this to hold on various settings with multiple assisted decoding candidates, highlighting its flexibility and the advantageous role that such decision making can play.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑