发表机构
University of California-San Diego; Microsoft Research（加州大学圣迭戈分校; 微软研究院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出约束感知训练目标，将程序分析外部化于训练过程，通过三个定理和合成实验证明其相比交叉熵训练在同等参数和数据下降低预测损失，提升模型效率。
AI 中文摘要
当使用语言模型生成程序时，约束解码可以应用程序分析来排除违反语法、作用域或类型规则的令牌。然而，存在一种重复：标准训练已经教会模型抑制这些分析所拒绝的令牌。这种重复引出一个问题：如果在推理过程中无论如何都要执行某种分析来过滤掉一组令牌，我们能否在训练期间完全避免教会模型该分析，并且这种外部化是否会导致更高效的模型？本文定义了一个满足这种外部化需求的通用约束感知目标，并将外部化的益处形式化为关于模型大小和数据效率的三个具体定理。我们通过一个受控的合成实验表明，这些定理在训练动态中仍然成立：与普通交叉熵训练相比，约束感知训练在匹配的参数数量和数据的条件下实现了更低的预测损失，从而激励了在训练目标中纳入生成过程中所使用的分析。
英文摘要
When generating programs with language models, constrained decoding can apply program analyses to exclude tokens that violate syntax, scope, or typing rules. However, there is a duplication: standard training already teaches the model to suppress the tokens rejected by these analyses. This duplication leads to the question: if we will perform some analysis to filter a set tokens out during inference anyways, can we avoid teaching the model the said analysis altogether during training, and does this externalization lead to more efficient models? This paper defines a general constraint-aware objective satisfying this externalization desideratum and formalizes the benefits of externalization into three concrete theorems about model size and data efficiency. We show, through a controlled synthetic experiment, that the theorems survive training dynamics: constraint-aware training yields lower prediction loss at a matched parameter count and data compared to ordinary cross-entropy training, motivating training objectives that incorporate the analyses used during generation.
CommentsAccepted at Neurips 2026 workshop AI for Verifiable Coding
