arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2110.04367cs.LGstat.ML
收藏

混合随机特征

Hybrid Random Features

Google Brain Robotics(谷歌大脑机器人学);Columbia University(哥伦比亚大学);Yale University(耶鲁大学);University of Cambridge(剑桥大学);The Alan Turing Institute(艾伦·图灵研究所)
Krzysztof Choromanski, Haoxian Chen, Han Lin, Yuanzhe Ma, Arijit Sehanobish, Deepali Jain, Michael S Ryoo, Jake Varley, Andy Zeng, Valerii Likhosherstov, Dmitry… 展开作者

Krzysztof Choromanski, Haoxian Chen, Han Lin, Yuanzhe Ma, Arijit Sehanobish, Deepali Jain, Michael S Ryoo, Jake Varley, Andy Zeng, Valerii Likhosherstov, Dmitry Kalashnikov, Vikas Sindhwani, Adrian Weller

摘要:

我们提出了一类新的随机特征方法,用于线性化softmax和高斯内核,称为混合随机特征(HRF),可以自动调整内核估计的质量,以在定义的感兴趣区域中提供最准确的近似。HRF的特殊实例导致众所周知的方法,例如三角函数(Rahimi和Recht,2007)或(最近在线性注意力Transformers的背景下引入的)正随机特征(Choromanski等人,2021年)。通过将Bochner定理推广到softmax/Gaussian内核,并利用组合内核的随机特性,HRF机制提供了强有力的理论保证-无偏近似和严格小于其对应物的最坏情况相对误差。我们进行详尽的经验评估HRF范围从逐点核估计实验,通过测试数据承认聚类结构的基准隐含注意Transformers(也为下游机器人应用),证明其质量在广泛的机器学习问题。

英文摘要:

We propose a new class of random feature methods for linearizing softmax and Gaussian kernels called hybrid random features (HRFs) that automatically adapt the quality of kernel estimation to provide most accurate approximation in the defined regions of interest. Special instantiations of HRFs lead to well-known methods such as trigonometric (Rahimi and Recht, 2007) or (recently introduced in the context of linear-attention Transformers) positive random features (Choromanski et al., 2021). By generalizing Bochner's Theorem for softmax/Gaussian kernels and leveraging random features for compositional kernels, the HRF-mechanism provides strong theoretical guarantees - unbiased approximation and strictly smaller worst-case relative errors than its counterparts. We conduct exhaustive empirical evaluation of HRF ranging from pointwise kernel estimation experiments, through tests on data admitting clustering structure to benchmarking implicit-attention Transformers (also for downstream Robotics applications), demonstrating its quality in a wide spectrum of machine learning problems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑