Review Beats Planning: 双模型交互模式用于代码合成
Review Beats Planning: Dual-Model Interaction Patterns for Code Synthesis
AI总结:
通过双模型交互模式提升代码生成性能,审查优于规划,增强规范质量可进一步提高效果。
摘要:
如何让两个语言模型交互以产生比单独一个更好的代码?传统的做法——推理模型计划,代码专家实现——似乎自然但失败了:在HumanEval+上,计划后生成的代码性能比单独代码专家下降2.4个百分点。我们展示反转交互改变了所有。当代码专家自由生成,推理模型不计划而进行审查时,相同的两个模型在相同硬件上达到90.2%的pass@1——超过GPT-4o(87.2%)和O1 Preview(89.0%)——在约2美元每小时的商用GPU上。跨基准验证在542个问题(HumanEval+和MBPP+)上揭示了一个调节变量:审查有效性与规范丰富性成正比,产生丰富规范问题的改进(+9.8个百分点)比瘦规范问题的改进(+2.3个百分点)多四倍,而在两种情况下都保持净正收益。实际意义是双重的:根据认知优势组合模型(审查者审查,编码者编码),并投资于规范质量以放大收益。
英文摘要:
How should two language models interact to produce better code than either can alone? The conventional approach -- a reasoning model plans, a code specialist implements -- seems natural but fails: on HumanEval+, plan-then-code degrades performance by 2.4 percentage points versus the code specialist alone. We show that reversing the interaction changes everything. When the code specialist generates freely and the reasoning model reviews instead of plans, the same two models on the same hardware achieve 90.2% pass@1 -- exceeding GPT-4o (87.2%) and O1 Preview (89.0%) -- on ~$2/hr of commodity GPU. Cross-benchmark validation across 542 problems (HumanEval+ and MBPP+) reveals a moderating variable: review effectiveness scales with specification richness, yielding 4x more improvement on richly-specified problems (+9.8pp) than on lean ones (+2.3pp), while remaining net-positive in both cases. The practical implication is twofold: compose models by their cognitive strengths (reviewers review, coders code), and invest in specification quality to amplify the returns.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。