CodeFuse-CommitEval: 向基于提交信息和代码变更不一致检测的LLM能力基准测试迈进
CodeFuse-CommitEval: Towards Benchmarking LLM's Power on Commit Message and Code Change Inconsistency Detection
AI总结：
CodeFuse-CommitEval是首个用于评估LLM检测提交信息与代码变更不一致能力的基准，通过生成七类不一致消息并验证样本，评估六种开源LLM在不同增强策略下的表现，揭示了不同增强方法对模型性能的影响及不同类型不一致的检测差异。
摘要：
版本控制依赖提交信息来传达代码变更的理由，但这些信息往往质量低下，更严重的是与它们的差异不一致——称为消息-代码不一致（MCI）。MCIs会误导审阅者，阻碍维护，污染研究数据集，并可能掩盖安全补丁。然而，目前没有专门的基准来评估MCI检测模型。我们介绍了CODEFUSE-COMMITEVAL，这是第一个使用大型语言模型（LLMs）进行MCI检测的基准。基于ApacheCM数据集的多样性和质量，我们通过规则引导的突变生成七种不一致的消息，并通过双重验证验证正负样本。使用这个标记的消息-差异对数据集，我们评估了六个最先进的开源LLMs，在常规设置和三种增强策略下：少样本提示、思维链和扩展上下文。结果显示，模型检测不一致提交比一致提交更可靠（平均召回率85.95%，精确率80.28%，特异度63.8%）；gpt-oss-20B总体表现最佳，但使用其他模型的两倍以上token。增强效果各异：相邻上下文有助于大模型但为小模型增加噪声；少样本提高准确性并减少token使用，但增加普遍错误预测；思维链提高精确率和特异度，但以召回率和更高的token消耗为代价。按类型分析显示，组件、文件路径和操作不一致的检测性更高，但意图层面“目的”不一致的准确性较低且token成本更高。CODEFUSE-COMMITEVAL为测量、比较和推进MCI检测提供了严谨的基础，突显了需要更丰富的上下文和平衡的数据来捕捉高层语义差距的必要性。
英文摘要：
Version control relies on commit messages to convey the rationale for code changes, but these messages are often low quality and, more critically, inconsistent with their diffs-known as message-code inconsistency (MCI). MCIs mislead reviewers, hinder maintenance, contaminate research datasets, and may obscure security patches. Yet, no dedicated benchmark exists to evaluate models for MCI detection. We introduce CODEFUSE-COMMITEVAL, the first benchmark designed for MCI detection using large language models (LLMs). Built on the ApacheCM dataset for diversity and quality, we generate seven types of inconsistent messages through rule-guided mutations of originally consistent commits and apply two-fold validation to verify both positive and negative samples. Using this labeled dataset of message-diff pairs, we evaluate six state-of-the-art open-source LLMs under a vanilla setting and with three augmentation strategies: few-shot prompting, chain-of-thought, and extended context. Results show models detect inconsistent commits more reliably than consistent ones (average Recall 85.95%, Precision 80.28%, Specificity 63.8%); gpt-oss-20B performs best overall but uses over twice the tokens of others. Augmentation effects vary: adjacent context helps larger models but adds noise for smaller ones; few-shot improves accuracy and reduces token use, yet increases universally incorrect predictions; chain-of-thought boosts precision and specificity at the cost of recall and higher token consumption. Type-wise analysis reveals higher detectability for component, file-path, and operation inconsistencies, but lower accuracy and higher token cost for intent-level "purpose" inconsistencies. CODEFUSE-COMMITEVAL provides a rigorous foundation for measuring, comparing, and advancing MCI detection, highlighting the need for richer context and balanced data to capture high-level semantic gaps.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
