发表机构
University of Edinburgh（爱丁堡大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究如何约束自回归大语言模型生成，核心方法是将其提炼为可处理概率模型，利用有限时长$LR(k)$文法可多项式时间计算满足情况的特性，实现对LLM生成的有效约束与引导，确保输出满足形式句法约束。
AI 中文摘要
约束自回归大语言模型（LLMs）的生成是将语言模型集成到形式系统中的重要组成部分。在程序合成等任务的代码和数据生成中，确保语言模型产生语法有效的输出是处理此类输出的前提条件。这些语言（如SQL或JSON）通常设计为$LR(k)$上下文无关文法。通过将LLM提炼为可处理的概率模型，其自回归生成可以被引导和屏蔽，以纳入满足逻辑约束的概率，确保高质量且有效的输出。本文证明了任何有限时长的$LR(k)$文法的满足情况都可以在多项式时间内计算出来，这比以前方法应用于此类文法的指数时间有所改进。该结果能够对LLM生成进行有效约束和引导，使其输出更好地满足形式句法约束。
英文摘要
Constraining the generation of autoregressive large language models (LLMs) is an important component of integrating language models into formal systems. In the generation of code and data for tasks like program synthesis, ensuring that language models produce syntactically valid output is a prerequisite for processing such output. These languages (such as SQL or JSON) are often designed as $LR(k)$ context-free grammars. By distilling the LLM to a tractable probabilistic model, its autoregressive generation can be steered and masked to incorporate the probability of satisfying logical constraints, ensuring high quality output that is guaranteed to be valid. This paper demonstrates that the satisfaction of any $LR(k)$ grammar of finite duration can be calculated in polynomial time, an improvement over the exponential time of applying previous methods to such grammars. This result enables efficient constraint and steering of LLM generation towards output that better satisfies formal syntactic constraints.
Comments21 pages, 4 figures, 2 algorithms
