通过低logit秩从现代语言模型中获得可证明的学习
Provably Learning from Modern Language Models via Low Logit Rank
AI总结：
本文提出了一种基于低logit秩结构的高效算法，用于从查询中获得现代语言模型的可证明学习保证。
摘要：
尽管现代语言模型及其内部机制极其复杂，最近的工作（Golowich, Liu & Shetty；2025）提出了一种简单且可能具有可处理性的抽象方法，通过观察到经验上这些语言模型似乎都具有近似低logit秩。大致来说，这意味着由模型对各种token的log概率组成的矩阵在某些token序列条件下可以近似为低秩矩阵。本文的重点是理解如何利用这种结构算法地获得可证明的学习保证。由于低logit秩模型可以编码难以学习的分布，如噪声奇偶性，我们研究一种具有logit查询的查询学习模型，该模型反映了常见API的访问模型。我们的主要结果是为从查询中学习任何近似低logit秩模型提供高效的算法。我们强调，我们的结构假设紧密反映了在现代语言模型中经验上观察到的行为。因此，我们的结果给出了我们相信是第一个为生成模型提供端到端学习保证的结果，该生成模型可能捕捉现代语言模型。
英文摘要：
While modern language models and their inner workings are incredibly complex, recent work (Golowich, Liu & Shetty; 2025) has proposed a simple and potentially tractable abstraction for them through the observation that empirically, these language models all seem to have approximately low logit rank. Roughly, this means that a matrix formed by the model's log probabilities of various tokens conditioned on certain sequences of tokens is well approximated by a low rank matrix. In this paper, our focus is on understanding how this structure can be exploited algorithmically for obtaining provable learning guarantees. Since low logit rank models can encode hard-to-learn distributions such as noisy parities, we study a query learning model with logit queries that reflects the access model for common APIs. Our main result is an efficient algorithm for learning any approximately low logit rank model from queries. We emphasize that our structural assumption closely reflects the behavior that is empirically observed in modern language models. Thus, our result gives what we believe is the first end-to-end learning guarantee for a generative model that plausibly captures modern language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
