TRACER: 一种用于代码大语言模型中细粒度污染检测的语义感知框架
TRACER: A Semantic-Aware Framework for Fine-Grained Contamination Detection in Code LLMs
AI总结:
提出TRACER框架,通过三级语义重叠和粗到细流水线检测代码LLM中的细粒度数据污染,在基准测试中F1达0.91。
摘要:
数据污染是对模型评估可靠性的已知威胁。然而,在代码大语言模型(LLM)中,污染往往超出精确重复,这一问题仍未得到充分探索。我们提出了TRACER,一种用于细粒度代码污染检测的语义感知框架。TRACER使用三级语义重叠——功能相同、几乎相同和共享逻辑——对污染进行建模,并通过粗到细的流水线进行检测。我们还引入了首个细粒度代码污染检测基准,涵盖三个广泛使用的基准和三个具有代表性的后训练数据集。TRACER在多个LLM骨干网络上取得了强大且一致的性能,其中GPT-5在细粒度检测中F1分数达到0.91。在二分类设置中,TRACER的F1达到0.92,比现有方法高出42%-217%。我们进一步进行了消融研究和错误分析,以评估TRACER中各个组件的贡献。
英文摘要:
Data contamination is a known threat to the reliability of model evaluation. However, it remains underexplored in code large language models (LLMs), where contamination often goes beyond exact duplication. We present TRACER, a semantic-aware framework for fine-grained code contamination detection. TRACER models contamination using three levels of semantic overlap - Functionally Identical, Nearly Identical, and Shared Logic - and detects them through a coarse-to-fine pipeline. We also introduce the first benchmark for fine-grained code contamination detection, spanning three widely used benchmarks and three representative post-training datasets. TRACER achieves strong and consistent performance across multiple LLM backbones, with GPT-5 reaching an F1 score of 0.91 in fine-grained detection. In the binary setting, TRACER attains an F1 of 0.92, outperforming existing methods by 42%-217%. We further conduct ablation studies and error analysis to assess the contributions of individual components in TRACER.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。