IntentCoding: 提升代码生成中用户意图的遵循能力
IntentCoding: Amplifying User Intent in Code Generation
AI总结:
IntentCoding通过增强用户意图的影响力,提升代码生成中对约束条件的遵循能力,显著提高约束满足和功能正确性。
摘要:
大型语言模型(LLMs)在代码生成方面展现了强大的能力,但其在面对多个约束条件下的用户意图的遵循仍是一个显著挑战。我们的实证分析揭示了两个关键观察:1)随着用户意图中约束条件数量的增加,模型性能迅速下降;2)虽然用户意图会影响模型的logits,但这种影响可能不足以有效引导解码过程。为此,我们提出了Intent-Amplified Code Generation(IntentCoding),一种新颖的解码策略,旨在增强LLM遵循用户意图的能力。IntentCoding通过屏蔽意图来捕捉用户意图的影响,并应用多强度集成机制在生成过程中放大用户意图的效果。IntentCoding具有模型无关性,不需要额外训练,并且可以无缝集成到现有的解码流程中。为了实现系统的评估,我们还构建了CodeConstraints,一个专门设计用于在不同约束数量下测试用户意图合规性的基准数据集。在我们构建的Constraints以及流行的IFEvalCode、HumanEval和LiveCodeBench数据集上的实验表明,我们的IntentCoding模型在约束满足和功能正确性方面相比标准解码方法有显著提升。IntentCoding在CodeConstraints上实现了高达71.0%的相对提升,在IFEvalCode上实现了高达67.3%的相对提升,在HumanEval和LiveCodeBench上的pass@1表现上实现了高达29.3%的提升,相比贪婪解码。
英文摘要:
Large Language Models (LLMs) have shown strong capabilities in code generation, but their adherence to fine-grained user intent with multiple constraints remains a significant challenge. Our empirical analysis reveals two key observations: 1) Model performance deteriorates quickly as the number of constraints in the user intent increases, and 2) While user intent does influence the model's logits, such an influence may not be strong enough to effectively steer the decoding process. To this end, we propose Intent-Amplified Code Generation (IntentCoding), a novel decoding strategy that enhances an LLM's ability to follow user intent. IntentCoding captures the influence of user intent by masking out the intent, and applies a multi-strength ensemble mechanism to amplify the effect of user intent during generation. IntentCoding is model-agnostic, requires no additional training, and integrates seamlessly with existing decoding procedures. To enable systematic evaluation, we also construct CodeConstraints, a benchmark dataset specifically designed to test user intent compliance under varying numbers of constraints. Experiments on our constructed Constraints, as well as popular IFEvalCode, HumanEval and LiveCodeBench datasets, show that our IntentCoding model significantly improves both constraint satisfaction and functional correctness compared to standard decoding approaches. IntentCoding achieves up to 71.0% relative improvement on CodeConstraints, achieves up to 67.3% relative improvement on IFEvalCode and achieves up to 29.3% relative improvement in pass@1 on HumanEval and LiveCodeBench compared with greedy decoding.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。