arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.12487cs.SEcs.LG
收藏

Diff-XYZ: 评估差异理解能力的基准

Diff-XYZ: A Benchmark for Evaluating Diff Understanding

JetBrains Research(JetBrains研究)

Evgeniy Glukhov, Michele Conti, Egor Bogomolov, Yaroslav Golubev, Alexander Bezzubov

AI总结:

为评估智能体处理代码差异的能力,本文提出Diff-XYZ基准(含应用、反向应用、差异生成三任务),基于真实提交数据,实证研究发现差异格式选择需结合任务和模型大小,为LLM代码编辑能力优化提供基础。

摘要:

可靠处理代码差异(code diffs)是大规模编辑和重构代码仓库的智能体的核心能力。我们引入Diff-XYZ,一个紧凑的代码差异理解基准,包含三个监督任务:应用(旧代码+差异→新代码)、反向应用(新代码-差异→旧代码)和差异生成(新代码-旧代码→差异)。基准中的实例是三元组⟨旧代码,新代码,差异⟩,来自CommitPackFT的真实提交,配有自动指标和清晰的评估协议。我们使用该基准对统一差异格式进行了聚焦实证研究,并对不同差异表示形式进行了跨格式比较。研究结果表明,应根据使用场景和模型大小选择不同格式:例如,在大多数任务中,较大模型使用搜索-替换格式表示差异的性能最佳,而结构化udiff变体性能相近但略弱;相比之下,较小的开源模型从任何格式选择中获益甚微。Diff-XYZ基准是评估和改进LLM差异处理能力的可重用基础,有助于未来差异格式和代码编辑模型的开发。数据集发布于HuggingFace Hub:https://huggingface.co/datasets/JetBrains-Research/diff-xyz。

英文摘要:

Reliable handling of code diffs is central to agents that edit and refactor repositories at scale. We introduce Diff-XYZ, a compact benchmark for code-diff understanding with three supervised tasks: apply (old code $+$ diff $\rightarrow$ new code), anti-apply (new code $-$ diff $\rightarrow$ old code), and diff generation (new code $-$ old code $\rightarrow$ diff). Instances in the benchmark are triples $\langle \textit{old code}, \textit{new code}, \textit{diff} \rangle$ drawn from real commits in CommitPackFT, paired with automatic metrics and a clear evaluation protocol. We use the benchmark to do a focused empirical study of the unified diff format and run a cross-format comparison of different diff representations. Our findings reveal that different formats should be used depending on the use case and model size. For example, representing diffs in search-replace format performs best for larger models across most tasks, while structured udiff variants offer similar but slightly weaker performance. In contrast, smaller open models benefit little from any formatting choice. The Diff-XYZ benchmark is a reusable foundation for assessing and improving diff handling in LLMs that can aid future development of diff formats and models editing code. The dataset is published on HuggingFace Hub: https://huggingface.co/datasets/JetBrains-Research/diff-xyz.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑