SWE-ABS: 通过对抗性基准测试揭示基于测试的基准测试中膨胀的成功率
SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark
AI总结：
SWE-ABS通过对抗性测试揭示测试基准中被高估的成功率，显著提升测试套件质量并降低顶级代理的得分。
摘要：
SWE-Bench Verified排行榜正接近饱和，其中顶级系统达到78.80%的性能。然而，我们证明这种性能是被高估的。我们的重新评估发现，top-30代理中有一份五的“解决”补丁是语义错误的，仅因为弱测试套件未能暴露其错误。我们提出了SWE-ABS，一种对抗性框架，通过两阶段流程加强测试套件：（1）基于覆盖的增强使用程序切片来针对未测试的代码区域，（2）基于突变的对抗性测试合成合理但错误的补丁以暴露语义盲点。在SWE-Bench Verified（500个实例）上，SWE-ABS加强了50.2%的实例，比先前工作提高了25.1倍，拒绝了19.71%的先前通过的补丁。结果，顶级代理的分数从78.80%降至62.20%，导致显著的排行榜重新排列，先前排名第一的代理跌至第五名。
英文摘要：
The SWE-Bench Verified leaderboard is approaching saturation, with the top system achieving 78.80%. However, we show that this performance is inflated. Our re-evaluation reveals that one in five "solved" patches from the top-30 agents are semantically incorrect, passing only because weak test suites fail to expose their errors. We present SWE-ABS, an adversarial framework that strengthens test suites through a two-stage pipeline: (1) coverage-driven augmentation using program slicing to target untested code regions, and (2) mutation-driven adversarial testing that synthesizes plausible but incorrect patches to expose semantic blind spots. On SWE-Bench Verified (500 instances), SWE-ABS strengthens 50.2% of instances, a 25.1x improvement over prior work, and rejects 19.71% of previously passing patches. As a result, the top agent's score decreases from 78.80% to 62.20%, leading to significant leaderboard reshuffling, with the previous top-ranked agent dropping to fifth place.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
