arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.11016cs.LGcs.CL
收藏

面向GPU硬件加速器的优化投机采样

Optimized Speculative Sampling for GPU Hardware Accelerators

Technische Hochschule Nürnberg Georg Simon Ohm(纽伦堡乔治·欧姆应用技术大学);KAIST(韩国科学技术院)

Dominik Wagner, Seanie Lee, Ilja Baumann, Philipp Seeberger, Korbinian Riedhammer, Tobias Bocklet

AI总结:

本文通过并发计算中间矩阵和用sigmoid近似softmax来优化GPU上的投机采样,在自动语音识别和摘要任务上实现6%到94%的加速,且准确性损失很小。

摘要:

在这项工作中,我们针对并行硬件加速器优化投机采样,以提高采样速度。我们注意到,投机采样所需中间矩阵的相当一部分可以并发计算。这使我们能够将工作负载分配到多个GPU线程上,从而在线程块内对矩阵片段进行同时操作。相对于基线实现,这带来了6%到13%的分析时间改进,且不牺牲准确性。为了进一步加速投机采样,由softmax参数化的概率分布用sigmoid进行近似。这种近似方法在分析时间上带来了显著更大的相对改进,范围从37%到94%,准确性仅有轻微下降。我们在自动语音识别和摘要任务上进行了大量实验,以验证我们优化方法的有效性。

英文摘要:

In this work, we optimize speculative sampling for parallel hardware accelerators to improve sampling speed. We notice that substantial portions of the intermediate matrices necessary for speculative sampling can be computed concurrently. This allows us to distribute the workload across multiple GPU threads, enabling simultaneous operations on matrix segments within thread blocks. This results in profiling time improvements ranging from 6% to 13% relative to the baseline implementation, without compromising accuracy. To further accelerate speculative sampling, probability distributions parameterized by softmax are approximated by sigmoid. This approximation approach results in significantly greater relative improvements in profiling time, ranging from 37% to 94%, with a minor decline in accuracy. We conduct extensive experiments on both automatic speech recognition and summarization tasks to validate the effectiveness of our optimization methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑