SWE-IF: 使代码评估与人类偏好对齐
SWE-IF: Aligning Code Evaluation with Human Preference
AI总结：
提出SWE-IF基准，通过可验证指令分类法VeriCode评估代码指令遵循能力，发现指令遵循是区分LLM代码质量的关键，与功能正确性结合更能匹配人类偏好。
摘要：
大型语言模型（LLM）推动了vibe coding，用户通过自然语言交互利用LLM生成并迭代优化代码，直到通过其vibe检查。Vibe检查反映了人类偏好，超越了功能性：解决方案应感觉正确、阅读清晰、保留意图并保持正确。然而，当前的代码评估仍局限于pass@k，仅捕获功能正确性，忽略了用户常规应用的非功能性指令。在本文中，我们假设指令遵循是vibe检查中除功能正确性之外缺失的部分。为了用量化信号衡量模型的代码指令遵循能力，我们提出了VeriCode，一个包含30条可验证代码指令及其确定性验证器的分类法。我们使用该分类法增强现有评估套件，得到SWE-IF，一个评估指令遵循和功能正确性的测试平台。评估31个LLM，我们发现即使最强的模型也难以遵守多条指令，并表现出功能回归。最重要的是，功能正确性和指令遵循的复合得分与人类偏好相关性最强，其中指令遵循成为LLM之间的主要区分因素。我们的代码、数据和分类法可在https://github.com/maszhongming/SWE-IF获取。
英文摘要：
Large Language Models (LLMs) have catalyzed vibe coding, where users leverage LLMs to generate and iteratively refine code through natural language interactions until it passes their vibe check. Vibe check reflects human preference and goes beyond functionality: the solution should feel right, read cleanly, preserve intent, and remain correct. However, current code evaluation remains anchored to pass@k and captures only functional correctness, overlooking non-functional instructions that users routinely apply. In this paper, we hypothesize that instruction following is the missing piece underlying vibe check besides functional correctness. To quantify models' code instruction-following capabilities with measurable signals, we present VeriCode, a taxonomy of 30 verifiable code instructions together with deterministic verifiers. We use the taxonomy to augment established evaluation suites, resulting in SWE-IF, a testbed to assess both instruction following and functional correctness. Evaluating 31 LLMs, we show that even the strongest models struggle to comply with multiple instructions and exhibit functional regression. Most importantly, a composite score of functional correctness and instruction following correlates best with human preference, with instruction following emerging as the primary differentiator among LLMs. Our code, data, and taxonomy are available at https://github.com/maszhongming/SWE-IF.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
