arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.06003stat.MLcs.LG
收藏

Softmax模型与Leverage Score模型的二元假设检验

Binary Hypothesis Testing for Softmax Models and Leverage Score Models

Institute for Advanced Study(高等研究院);University of California Berkeley(加州大学伯克利分校);Boston University(波士顿大学)

Yuzhou Gu, Zhao Song, Junze Yin

AI总结:

本文将LLM中的注意力单元抽象为softmax模型,研究其二元假设检验问题,证明样本复杂度渐近为$O(ε^{-2})$,并将该分析方法类比推广至线性代数中的leverage score模型,获得相似结论。

摘要:

Softmax分布广泛应用于机器学习,包括大型语言模型(LLMs),其中注意力单元使用了softmax分布。我们将注意力单元抽象为softmax模型,其中给定一个向量输入,模型产生一个从softmax分布(依赖于该向量输入)中抽取的输出。我们考虑了在softmax模型设定下的二元假设检验这一基本问题。即,给定一个未知的softmax模型,且已知它是两个给定softmax模型之一,需要多少次查询才能确定哪一个是真实的?我们证明了样本复杂度渐近为$O(ε^{-2})$,其中$ε$是模型参数之间的某种距离。此外,我们在softmax模型与leverage score模型之间进行了类比,后者是线性代数和图论中算法设计的重要工具。在较高层面上,leverage score模型是一种给定向量输入时,产生一个从依赖于该输入的分布中抽取的输出的模型。对于leverage score模型的二元假设检验问题,我们获得了类似的结果。

英文摘要:

Softmax distributions are widely used in machine learning, including Large Language Models (LLMs), where the attention unit uses softmax distributions. We abstract the attention unit as the softmax model, where given a vector input, the model produces an output drawn from the softmax distribution (which depends on the vector input). We consider the fundamental problem of binary hypothesis testing in the setting of softmax models. That is, given an unknown softmax model, which is known to be one of the two given softmax models, how many queries are needed to determine which one is the truth? We show that the sample complexity is asymptotically $O(ε^{-2})$ where $ε$ is a certain distance between the parameters of the models. Furthermore, we draw an analogy between the softmax model and the leverage score model, an important tool for algorithm design in linear algebra and graph theory. The leverage score model, on a high level, is a model which, given a vector input, produces an output drawn from a distribution dependent on the input. We obtain similar results for the binary hypothesis testing problem for leverage score models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑