基于自动机的语言模型解码约束
Automata-based constraints for language model decoding
AI总结：
针对语言模型难以保证生成形式语言有效字符串的问题，本文用自动机理论重构解码约束，为正则语言和确定性上下文无关语言给出高效闭式解，实现约7,000倍加速、可证明正确性与模块化扩展。
摘要：
语言模型（LM）通常被期望生成某种形式语言中的字符串；例如结构化数据、API调用或代码片段。尽管可以通过调优来提高LM对形式语法的遵从性，但这并不能保证一致性，尤其是在适合大规模部署的较小LM上。此外，调优需要大量资源，这使其对于不常见或任务特定的格式而言并不现实。为了防止下游解析错误，理想情况下我们希望约束LM只生成有效输出，但分词化使这一问题变得极为复杂，因为分词通常既具有歧义，又与形式语法不对齐。我们通过应用自动机理论解决了这些问题，为正则语言推导了一种高效的闭式解；正则语言是一类具有许多实际应用的广泛形式语言，包括API调用或模式引导的JSON和YAML。我们还讨论了应对高分支因子问题的实用扩展，并将技术扩展到确定性上下文无关语言，这类语言同样允许高效的闭式解。此前关于该主题的工作（Willard和Louf，2023）将定制化解决方案叠加在自动机之上，导致速度、正确性和可扩展性方面的问题。相反，我们用自动机重新表述整个任务，从而能够利用经过充分研究和优化的算法。我们的系统编译约束的速度约快7,000倍，具有可证明的正确性，并能以模块化方式扩展。
英文摘要：
Language models (LMs) are often expected to generate strings in some formal language; for example, structured data, API calls, or code snippets. Although LMs can be tuned to improve their adherence to formal syntax, this does not guarantee conformance, especially with smaller LMs suitable for large-scale deployment. In addition, tuning requires significant resources, making it impractical for uncommon or task-specific formats. To prevent downstream parsing errors we would ideally constrain the LM to only produce valid output, but this is severely complicated by tokenization, which is typically both ambiguous and misaligned with the formal grammar. We solve these issues through the application of automata theory, deriving an efficient closed-form solution for the regular languages, a broad class of formal languages with many practical applications, including API calls or schema-guided JSON and YAML. We also discuss pragmatic extensions for coping with the issue of high branching factor, and extend our techniques to deterministic context-free languages, which similarly admit an efficient closed-form solution. Previous work on this topic (Willard and Louf, 2023) layers bespoke solutions onto automata, leading to problems with speed, correctness, and extensibility. Instead, we reformulate the entire task in terms of automata so we can leverage well-studied and well-optimized algorithms. Our system compiles constraints ~7,000x faster, is provably correct, and can be extended in a modular fashion.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
