arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.04076cs.SE
收藏

揭示大语言模型针对Java代码片段的真实类型推断能力

Unmasking the Genuine Type Inference Capabilities of LLMs for Java Code Snippets

Yiwen Dong, Zhenyang Xu, Yongqiang Tian, Chengnian Sun

备注：
under review
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究针对现有类型推断基准可能存在的数据泄露问题，构建了全新无泄露基准ThaliaType，验证了主流LLMs在真实场景下性能大幅下降，且受数据泄露影响的结果泛化性有限，为LLM代码类型推断能力评估提供了可靠基准。

摘要：

类型推断对复用在线代码片段至关重要。尽管代码片段在StackOverflow等平台上被广泛分享，但它们往往缺少关键的类型信息，例如完全限定名（FQNs）。近期研究已利用大语言模型（LLMs）为此类代码片段执行类型推断，展现出可观的结果。然而，这些结果可能存在数据泄露问题，因为用于评估的基准StatType-SO自2017年起就已在GitHub上公开。因此，目前仍不确定LLMs的优异表现反映的是对代码的真实语义理解，还是因为训练集中包含了真实标注数据。\n本文旨在全面评估LLMs在Java代码片段上的真实类型推断能力，并识别LLMs的潜在局限性。首先，我们构建了ThaliaType，这是一套全新的、此前未公开的基准套件，专为类型推断评估设计。其次，我们以StarCoder2 LLM为基线，发现StarCoder2的开源训练集中存在来自StatType-SO的数据泄露，并且观察到其他前沿LLMs在ThaliaType上评估时也出现类似的性能下降，其中精确率最高下降59%，召回率最高下降72%。最后，我们设计了保语义代码变换，以测试LLMs对代码片段执行语义的理解能力。结果表明，LLMs在StatType-SO上的性能对这些变换的鲁棒性远低于在ThaliaType上的表现，这说明其在StatType-SO上的性能可能受数据泄露影响而存在偏差，且泛化能力有限。\n这些发现凸显了精心设计的、无数据泄露的基准对于评估LLMs类型推断任务能力的重要性。我们建议未来研究采用ThaliaType，以严格、可靠地评估LLMs的真实类型推断能力。

英文摘要：

Type inference is crucial for reusing online code snippets. Although snippets are prevalently shared on platforms like StackOverflow, they often lack essential type information, such as fully qualified names (FQNs). Recent studies have leveraged Large Language Models (LLMs) to perform type inference for such code snippets, showing promising results. However, these results may suffer from data leakage, as the benchmark, StatType-SO, used for evaluation has been publicly available on GitHub since 2017. Consequently, it remains uncertain whether the strong performance of LLMs reflects genuine semantic understanding of code or is due to the ground truth being included in the training set. This paper strives to comprehensively evaluate the genuine type inference capabilities of LLMs on Java code snippets and identify potential limitations of LLMs. First, we created ThaliaType, a new, previously unreleased benchmark suite designed for type inference evaluation. Second, using the StarCoder2 LLM as baseline, we uncovered data leakage from StatType-SO in StarCoder2's open-source training set and observed that other state-of-the-art LLMs exhibit similar performance drops when evaluated on ThaliaType, with precision decreasing by up to 59% and recall by up to 72%. Finally, we designed semantic-preserving code transformations to test the capabilities of LLMs in understanding the execution semantics of snippets. Results showed that LLMs' performance on StatType-SO is far less robust to these transformations than on ThaliaType, suggesting that the performance on StatType-SO may be biased by data leakage and have limited generalizability. These findings highlight the importance of carefully designed, leakage-free benchmarks for evaluating LLMs on type inference tasks. We recommend future studies adopt ThaliaType for rigorous and reliable assessments of LLMs' genuine type inference capabilities.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑