arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.07978cs.DScs.CLcs.LG
收藏

无通信耦合与生成器不变推测解码

Coupling without Communication and Drafter-Invariant Speculative Decoding

New York University(纽约大学);Google Research, NY(谷歌研究)

Majid Daliri, Christopher Musco, Ananda Theertha Suresh

AI总结:

针对Alice和Bob需无通信使各自样本尽可能相等的问题,提出基于Gumbel采样的协议实现帕累托改进,并将其应用于推测解码构建生成器不变方案,实验中Gumbel采样优于加权MinHash。

摘要:

假设Alice拥有分布$P$,Bob拥有分布$Q$。Alice希望抽取样本$a\thicksim P$,Bob抽取样本$b\thicksim Q$,使$a=b$的概率尽可能高。已知通过从分布间的最优耦合中采样,Alice和Bob可实现$\text{Pr}[a=b]=1-D_{TV}(P,Q)$,其中$D_{TV}(P,Q)$是$P$与$Q$的总变差距离。若Alice和Bob必须完全无通信地解决同一问题呢?意外的是,借助公共随机性,他们可通过基于加权MinHash算法的简单协议,仍实现$\text{Pr}[a=b]\ngeq\frac{1-D_{TV}(P,Q)}{1+D_{TV}(P,Q)}\ngeq1-2D_{TV}(P,Q)$。[Bavarian等人,2020]已证明此界在最坏情况下是最优的。本文重新研究无通信耦合问题,为[Bavarian等人,2020]的最优性结果提供更简洁的证明。我们发现,虽然加权MinHash的最坏情况成功概率无法提升,但基于Gumbel采样的同等简单协议可实现帕累托改进:对任意分布对$P,Q$,Gumbel采样实现的$\text{Pr}[a=b]$均不低于加权MinHash。重要的是,这种改进可应用于实践。我们展示了无通信耦合在推测解码(一种近期用于加速自回归大语言模型的方法[Leviathan, Kalman, Matias, ICML 2023])中的应用。我们证明,无通信协议可用于构建生成器不变推测解码(\text{CSD})方案,其理想特性是:给定固定随机种子,无论使用何种生成器进行推测,输出均固定。在语言生成任务的实验中,Gumbel采样优于加权MinHash。代码可在https://github.com/majid-daliri/DISD获取。

英文摘要:

Suppose Alice has a distribution $P$ and Bob has a distribution $Q$. Alice wants to draw a sample $a\sim P$ and Bob a sample $b \sim Q$ such that $a = b$ with as high of probability as possible. It is well-known that, by sampling from an optimal coupling between the distributions, Alice and Bob can achieve $\Pr[a = b] = 1 - D_{TV}(P,Q)$, where $D_{TV}(P,Q)$ is the total variation distance between $P$ and $Q$. What if Alice and Bob must solve this same problem \emph{without communicating at all?} Perhaps surprisingly, with access to public randomness, they can still achieve $\Pr[a = b] \geq \frac{1 - D_{TV}(P,Q)}{1 + D_{TV}(P,Q)} \geq 1-2D_{TV}(P,Q)$ using a simple protocol based on the Weighted MinHash algorithm. This bound was shown to be optimal in the worst-case by [Bavarian et al., 2020]. In this work, we revisit the communication-free coupling problem. We provide a simpler proof of the optimality result from [Bavarian et al., 2020]. We show that, while the worst-case success probability of Weighted MinHash cannot be improved, an equally simple protocol based on Gumbel sampling offers a Pareto improvement: for every pair of distributions $P, Q$, Gumbel sampling achieves an equal or higher value of $\Pr[a = b]$ than Weighted MinHash. Importantly, this improvement translates to practice. We demonstrate an application of communication-free coupling to \emph{speculative decoding}, a recent method for accelerating autoregressive large language models [Leviathan, Kalman, Matias, ICML 2023]. We show that communication-free protocols can be used to contruct \emph{\CSD{}} schemes, which have the desirable property that their output is fixed given a fixed random seed, regardless of what drafter is used for speculation. In experiments on a language generation task, Gumbel sampling outperforms Weighted MinHash. Code is available at https://github.com/majid-daliri/DISD.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑