一致还是敏感？面向语义保持扰动的自动代码修订工具
Consistent or Sensitive? Automated Code Revision Tools Against Semantics-Preserving Perturbations
AI总结：
本文研究了自动代码修订工具在面对语义等价代码变体时的一致性问题，发现其生成正确修订的能力可能下降高达45.3%，并探讨了缓解策略的局限性。
摘要：
自动代码修订（ACR）工具旨在通过根据审阅者反馈自动生成代码修订来减少手动工作量。尽管ACR工具在历史数据上表现出色，但其在现实中的实用性取决于其处理相似代码变体的能力——即一致性。然而，ACR工具的概率性质往往会损害一致性，这可能导致即使语义等价的代码变体也会产生分歧的修订。在本文中，我们研究了ACR工具在面对语义等价的代码变体时保持一致性的程度。为此，我们首先设计了九种语义保持扰动（SPP），并将它们应用于2032个来自真实世界GitHub项目的Java方法，生成超过10,000个扰动变体用于评估。然后我们使用这些扰动来评估五种最先进的基于Transformer的ACR工具的一致性。我们发现，当面对语义等价的代码时，ACR工具生成正确修订的能力可能下降高达45.3%。扰动越接近目标区域，ACR工具生成正确修订的可能性就越低。我们探索了潜在的缓解策略，这些策略修改输入表示，但发现这些注意力引导的启发式方法只能带来微小的改进，因此将此问题的解决方案留作开放研究问题。
英文摘要：
Automated Code Revision (ACR) tools aim to reduce manual effort by automatically generating code revisions based on reviewer feedback. While ACR tools have shown promising performance on historical data, their real-world utility depends on their ability to handle similar code variants expressing the same issue - a property we define as consistency. However, the probabilistic nature of ACR tools often compromises consistency, which may lead to divergent revisions even for semantically equivalent code variants. In this paper, we investigate the extent to which ACR tools maintain consistency when presented with semantically equivalent code variants. To do so, we first designed nine types of semantics-preserving perturbations (SPP) and applied them to 2032 Java methods from real-world GitHub projects, generating over 10K perturbed variants for evaluation. Then we used these perturbations to evaluate the consistency of five state-of-the-art transformer-based ACR tools. We found that the ACR tools' ability to generate correct revisions can drop by up to 45.3%, when presented with semantically equivalent code. The closer the perturbation is to this targeted region, the more likely an ACR tool is to fail to generate the correct revision. We explored potential mitigation strategies that modify the input representation, but found that these attention-guiding heuristics yielded only marginal improvements, thus leaving the solution to this problem as an open research question.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
