Attention computation takes both the time complexity of $O(n^2)$ and the space complexity of $O(n^2)$ simultaneously, which makes deploying Large Language Models (LLMs) in streaming applications that involve long contexts requiring substantial computational resources. In recent OpenAI DevDay (Nov 6, 2023), OpenAI released a new model that is able to support a 128K-long document, in our paper, we focus on the memory-efficient issue when context length $n$ is much greater than 128K ($n \gg 2^d$). Considering a single-layer self-attention with Query, Key, and Value matrices $Q, K, V \in \mathbb{R}^{n \times d}$, the polynomial method approximates the attention output $T \in \mathbb{R}^{n \times d}$. It accomplishes this by constructing $U_1, U_2 \in \mathbb{R}^{n \times t}$ to expedite attention ${\sf Attn}(Q, K, V)$ computation within $n^{1+o(1)}$ time executions. Despite this, computing the approximated attention matrix $U_1U_2^\top \in \mathbb{R}^{n \times n}$ still necessitates $O(n^2)$ space, leading to significant memory usage. In response to these challenges, we introduce a new algorithm that only reads one pass of the data in a streaming fashion. This method employs sublinear space $o(n)$ to store three sketch matrices, alleviating the need for exact $K, V$ storage. Notably, our algorithm exhibits exceptional memory-efficient performance with super-long tokens. As the token length $n$ increases, our error guarantee diminishes while the memory usage remains nearly constant. This unique attribute underscores the potential of our technique in efficiently handling LLMs in streaming applications.
TCUQ: Single-Pass Uncertainty Quantification from Temporal Consistency with Streaming Conformal Calibration for TinyML
Multidisciplinary Faculty of Nador University Mohammed Premier Oujda(穆罕默德·布雷德·奥杰达大学多学科学院);Laboratory LaSTI, ENSAK Sultan Moulay Slimane University Khouribga(库尔希格大学拉斯特伊实验室)
Quick Streaming Algorithms for Maximization of Monotone Submodular Functions in Linear Time
Florida State University(佛罗里达州立大学)
考虑基数约束$k$下基集大小为$n$的单调次模极大化问题.对于这个问题,我们介绍了第一个确定性算法的线性时间复杂度,这些算法是流算法。对于任何$c \ge 1$,我们的单遍算法在$\lceil n / c \rceil + c$ Oracle查询中获得恒定的比率。此外,我们提出了一个确定性的,多遍流算法,通过一个恒定的数量,达到接近线性查询和时间复杂度的最佳比例。我们证明了一个下界,这意味着没有常数因子近似存在使用$o(n)$查询,即使查询不可行集是允许的。实证分析表明,我们的算法需要更少的查询(通常大大低于$n$),但仍然实现更好的客观价值比目前的国家的最先进的算法,包括单通,多通,和非流算法。
The Best Arm Evades: Near-optimal Multi-pass Streaming Lower Bounds for Pure Exploration in Multi-armed Bandits
Cheriton School of Computer Science, University of Waterloo(滑铁卢大学切尔顿计算机科学学院);Rutgers University(罗格斯大学);Rice University(莱斯大学);Texas A&M University(德克萨斯农工大学);Google Research(谷歌研究院)
Faster Sublinear Algorithms using Conditional Sampling
MIT(麻省理工学院)
概率分布D的条件采样预言机返回来自限制于域的指定子集的D的条件分布的样本。最近的一系列工作(Chakraborty et al. 2013和Cannone et al. 2014)表明,访问这样的条件抽样oracle只需要多对数甚至常数数量的样本来解决分布测试问题,如同一性和均匀性。这大大改善了标准采样模型,其中多项式许多样本是必要的。受这些结果的启发,我们引入了一个计算模型的基础上,条件采样开发次线性算法与指数更快的运行时间相比,标准的次线性算法。本文主要研究高维欧氏空间中点集上的几何优化问题。访问这些点是通过一个有条件的采样预言机提供的,该预言机将域的子集的简洁表示作为输入,并输出该子集中的均匀随机点。我们研究了两个很好的研究问题:k-均值聚类和估计的最小生成树的重量。与经典模型的已有算法相比,我们的算法具有时间、空间和样本复杂度,即维数为多项式,点数为多对数。最后,我们评论了该模型的适用性,并与现有的流,并行和分布式计算模型。