arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2205.15317cs.LGcs.AI
收藏

Chefs' Random Tables:非三角随机特征

Chefs' Random Tables: Non-Trigonometric Random Features

University of Cambridge(剑桥大学);Google Research(谷歌研究院);Columbia University(哥伦比亚大学);The Alan Turing Institute(阿兰·图灵研究所)

Valerii Likhosherstov, Krzysztof Choromanski, Avinava Dubey, Frederick Liu, Tamas Sarlos, Adrian Weller

摘要:

我们介绍厨师的随机表(CRT),一类新的非三角随机特征(RF)近似高斯和softmax内核。CRT是标准随机厨房水槽(RKS)方法的替代方案,该方法本质上依赖于三角图。我们提出的变体的CRT的RF是积极的,在最近的低秩Transformers的应用程序的一个关键要求。通过利用易于计算的统计数据,可以进一步减少方差。CRT的一个实例,最佳正随机特征(OPRF),是我们所知的第一个RF方法,用于无偏softmax内核估计,具有正的和有界的RF,导致指数小的尾部和比其对应物低得多的方差。正如我们所示,正交随机特征应用在OPRF提供额外的方差减少任何维度$d$(不仅渐近足够大的$d$,为RKS)。我们在许多任务上测试CRT,从非参数分类到针对文本、语音和图像数据训练Transformers,为低秩文本Transformers获得新的最先进的结果,同时提供线性空间和时间复杂度。

英文摘要:

We introduce chefs' random tables (CRTs), a new class of non-trigonometric random features (RFs) to approximate Gaussian and softmax kernels. CRTs are an alternative to standard random kitchen sink (RKS) methods, which inherently rely on the trigonometric maps. We present variants of CRTs where RFs are positive, a key requirement for applications in recent low-rank Transformers. Further variance reduction is possible by leveraging statistics which are simple to compute. One instantiation of CRTs, the optimal positive random features (OPRFs), is to our knowledge the first RF method for unbiased softmax kernel estimation with positive and bounded RFs, resulting in exponentially small tails and much lower variance than its counterparts. As we show, orthogonal random features applied in OPRFs provide additional variance reduction for any dimensionality $d$ (not only asymptotically for sufficiently large $d$, as for RKS). We test CRTs on many tasks ranging from non-parametric classification to training Transformers for text, speech and image data, obtaining new state-of-the-art results for low-rank text Transformers, while providing linear space and time complexity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑