The success of large language models for code relies on vast amounts of code data, including public open-source repositories, such as GitHub, and private, confidential code from companies. This raises concerns about intellectual property compliance and the potential unauthorized use of license-restricted code. While membership inference (MI) techniques have been proposed to detect such unauthorized usage, their effectiveness can be undermined by semantically equivalent code transformation techniques, which modify code syntax while preserving semantic.
In this work, we systematically investigate whether semantically equivalent code transformation rules might be leveraged to evade MI detection. The results reveal that model accuracy drops by only 1.5% in the worst case for each rule, demonstrating that transformed datasets can effectively serve as substitutes for fine-tuning. Additionally, we find that one of the rules (RenameVariable) reduces MI success by 10.19%, highlighting its potential to obscure the presence of restricted code. To validate these findings, we conduct a causal analysis confirming that variable renaming has the strongest causal effect in disrupting MI detection. Notably, we find that combining multiple transformations does not further reduce MI effectiveness. Our results expose a critical loophole in license compliance enforcement for training large language models for code, showing that MI detection can be substantially weakened by transformation-based obfuscation techniques.
DVD: A Robust Method for Detecting Variant Contamination in Large Language Model Evaluation
Beihang University(北京航空航天大学);Peking University(北京大学);Beijing University of Posts and Telecommunications(北京邮电大学);Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学)
QUITE: A Query Rewrite System Beyond Rules with LLM Agents
Sichuan University(四川大学);Cornell University(康奈尔大学);Purdue University(普渡大学);Hong Kong University of Science and Technology(香港科学与技术大学);Computer Network Information Center, Chinese Academy of Sciences(中国科学院计算机网络信息中心)
Knowlesdge Lab, University of Chicago, Chicago, Illinois, USA(知识实验室,芝加哥大学,芝加哥,伊利诺伊州,美国);Knowledge Lab, Data Science Institute, Department of Sociology, University of Chicago, Chicago, Illinois, USA(知识实验室,数据科学学院,社会学系,芝加哥大学,芝加哥,伊利诺伊州,美国)
University of Southern California(南加州大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);The Pennsylvania State University(宾夕法尼亚州立大学);University of California, Davis(加利福尼亚大学戴维斯分校)