紧凑约束编码用于LLM代码生成:关于令牌经济学和约束合规性的实证研究
Compact Constraint Encoding for LLM Code Generation: An Empirical Study of Token Economics and Constraint Compliance
AI总结:
本文通过实验证明紧凑约束头可减少令牌消耗而不影响约束合规性,发现编码形式和传播模式对合规率无显著影响,但约束类型和任务领域对合规性有显著影响。
摘要:
用于代码生成的LLM通常由工程约束指导,这些约束以冗长的自然语言表达。我们研究紧凑、结构化的约束头是否能减少提示令牌消耗而不影响约束合规性。在六个实验轮次中,涵盖11个模型、16个基准任务和超过830次LLM调用,发现紧凑头可减少约束部分令牌约71%,完整提示令牌25-30%。然而,三种编码形式和四种传播模式下约束满足率(CSR)无显著差异,观察到的效果大小微乎其微(Cliff's δ < 0.01,95%置信区间跨度±2.6个百分点)。这种无效应在不同能力层级的两个模型中均成立。补充实验显示非CSS任务提供了额外的跨领域支持。最大的合规性差异源在于约束类型(正常与反直觉约束之间Δ=9个百分点)和任务领域:反直觉约束对抗模型默认设置失败率10-100%,而常规约束在不同编码下均达到99%以上合规性。模型自我评估系统性高估合规性,揭示了约束理解与执行之间的差距。在测试条件下,紧凑约束编码的主要好处是减少令牌消耗而非提高合规性,工程努力应更侧重于约束设计而非提示格式。
英文摘要:
LLMs used for code generation are typically guided by engineering constraints--technology choices, dependency restrictions, and architectural patterns--expressed in verbose natural language. We investigate whether compact, structured constraint headers can reduce prompt token consumption without degrading constraint compliance. Across six experimental rounds spanning 11 models, 16 benchmark tasks, and over 830 LLM invocations, we find that compact headers reduce constraint-portion tokens by approximately 71% and full-prompt tokens by 25--30%, replicated across three independent rounds. However, we detect no statistically significant differences in constraint satisfaction rate (CSR) across three encoding forms or four propagation modes; observed effect sizes are negligible (Cliff's $δ$ < 0.01, 95% CI spanning $\pm$2.6 percentage points). This null pattern holds across two models from different capability tiers. A supplementary experiment with four non-CSS tasks provides additional cross-domain support for the encoding null result. The largest observed sources of compliance variance are constraint type ($Δ$ = 9 percentage points between normal and counter-intuitive constraints) and task domain: counter-intuitive constraints opposing model defaults fail at 10--100%, while conventional constraints achieve 99%+ compliance regardless of encoding. Model self-assessments systematically overestimate compliance relative to rule-based scoring, revealing a gap between constraint understanding and execution. Under the tested conditions, the primary benefit of compact constraint encoding is token reduction rather than compliance improvement, and engineering effort toward compliance is better directed at constraint design than prompt formatting.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。