Recently, large language models (LLMs) have garnered widespread attention for their exceptional capabilities. Prompts are central to the functionality and performance of LLMs, making them highly valuable assets. The increasing reliance on high-quality prompts has driven significant growth in prompt services. However, this growth also expands the potential for prompt leakage, increasing the risk that attackers could replicate original functionalities, create competing products, and severely infringe on developers' intellectual property. Despite these risks, prompt leakage in real-world prompt services remains underexplored.
In this paper, we present PRSA, a practical attack framework designed for prompt stealing. PRSA infers the detailed intent of prompts through very limited input-output analysis and can successfully generate stolen prompts that replicate the original functionality. Extensive evaluations demonstrate PRSA's effectiveness across two main types of real-world prompt services. Specifically, compared to previous works, it improves the attack success rate from 17.8% to 46.1% in prompt marketplaces and from 39% to 52% in LLM application stores, respectively. Notably, in the attack on "Math", one of the most popular educational applications in OpenAI's GPT Store with over 1 million conversations, PRSA uncovered a hidden Easter egg that had not been revealed previously. Besides, our analysis reveals that higher mutual information between a prompt and its output correlates with an increased risk of leakage. This insight guides the design and evaluation of two potential defenses against the security threats posed by PRSA. We have reported these findings to the prompt service vendors, including PromptBase and OpenAI, and actively collaborate with them to implement defensive measures.
BESA: Boosting Encoder Stealing Attack with Perturbation Recovery
School of Cyberspace Science and Technology, Beijing Institute of Technology(网络安全科学与技术学院,北京理工大学);Beijing Institute of Technology(北京理工大学);Singapore University of Technology and Design(新加坡科技设计大学)
Isolation and Induction: Training Robust Deep Neural Networks against Model Stealing Attacks
Beihang University(北京航空航天大学);Institute of Dataspace(数据空间研究院);Chinese Academy of Sciences(中国科学院);Sensetime Group Limited(商汤集团有限公司);Zhongguancun Laboratory(中关村实验室)
I Stolenly Swear That I Am Up to (No) Good: Design and Evaluation of Model Stealing Attacks
Christian Doppler Laboratory for Assurance;Transparency in Software Protection, University of Vienna Vienna Austria;SBA Research \& TU Wien Vienna Austria;IBM Research Zurich Zurich Switzerland;Institute of Information Systems Engineering, TU Wien Vienna Austria;Transparency in Software Protection, University of Vienna;SBA Research \& TU Wien;IBM Research Zurich;Institute of Information Systems Engineering, TU Wien
King Abdullah University of Science and Technology(卡塔尔国王 Abdullah 科学与技术大学);HitoX;University of Science and Technology of China(中国科学技术大学);Zhejiang University(浙江大学)
Probabilistically Robust Watermarking of Neural Networks
Artificial Intelligence Research Institute(人工智能研究所);Skolkovo Institute of Science and Technology(斯科尔科沃科学技术研究所);ISP RAS Research Center for Trusted Artificial Intelligence(俄罗斯科学院信息学研究所可信人工智能研究中心)
Quantifying (Hyper) Parameter Leakage in Machine Learning
Indraprastha Institute of Information Technology, Delhi, India(印度理工学院德里分校(Indraprastha Institute of Information Technology, Delhi, India));Institute for Systems Studies and Analyses, Delhi, India(系统研究与分析研究所(Institute for Systems Studies and Analyses, Delhi, India));Institute for Systems Studies(系统研究研究所(Institute for Systems Studies))