求解张量低循环秩逼近
Solving Tensor Low Cycle Rank Approximation
AI总结:
本文针对概率上下文无关文法中计算概率所涉及的张量循环秩问题,推广了“旋转与草图”技术,提出了一个输入稀疏时间算法,用于求解三阶张量的低循环秩逼近。
摘要:
大型语言模型已在现代生活中无处不在,广泛应用于自然语言处理、语言翻译和语音识别等多个领域。近期,一项突破性工作 [Zhao, Panigrahi, Ge, and Arora Arxiv 2023] 从概率上下文无关文法(PCFG)的角度解释了注意力模型。在 PCFG 中计算概率的核心计算任务之一,是构造一个特定的张量低秩逼近问题,我们可称之为张量循环秩。给定一个 $n \times n \times n$ 的三阶张量 $A$,若存在三个 $n \times k^2$ 大小的矩阵 $U$、$V$ 和 $W$,使得对于每个条目,对所有 $a \in [n], b \in [n], c \in [n]$,都有 \begin{align*} A_{a,b,c} = \sum_{i=1}^k \sum_{j=1}^k \sum_{l=1}^k U_{a,i+k(j-1)} \otimes V_{b, j + k(l-1)} \otimes W_{c, l + k(i-1) } \end{align*},则称 $A$ 具有循环秩-$k$。对于张量经典秩、Tucker 秩和 train 秩,[Song, Woodruff, Zhong SODA 2019] 已进行了充分研究。在本文中,我们推广了 [Song, Woodruff, Zhong SODA 2019] 第 186 页中先前的“旋转与草图”技术,并展示了一种针对循环秩的输入稀疏时间算法。
英文摘要:
Large language models have become ubiquitous in modern life, finding applications in various domains such as natural language processing, language translation, and speech recognition. Recently, a breakthrough work [Zhao, Panigrahi, Ge, and Arora Arxiv 2023] explains the attention model from probabilistic context-free grammar (PCFG). One of the central computation task for computing probability in PCFG is formulating a particular tensor low rank approximation problem, we can call it tensor cycle rank. Given an $n \times n \times n$ third order tensor $A$, we say that $A$ has cycle rank-$k$ if there exists three $n \times k^2$ size matrices $U , V$, and $W$ such that for each entry in each \begin{align*} A_{a,b,c} = \sum_{i=1}^k \sum_{j=1}^k \sum_{l=1}^k U_{a,i+k(j-1)} \otimes V_{b, j + k(l-1)} \otimes W_{c, l + k(i-1) } \end{align*} for all $a \in [n], b \in [n], c \in [n]$. For the tensor classical rank, tucker rank and train rank, it has been well studied in [Song, Woodruff, Zhong SODA 2019]. In this paper, we generalize the previous ``rotation and sketch'' technique in page 186 of [Song, Woodruff, Zhong SODA 2019] and show an input sparsity time algorithm for cycle rank.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。