The increasing reliance on large language models (LLMs) such as ChatGPT in various fields emphasizes the importance of ``prompt engineering,'' a technology to improve the quality of model outputs. With companies investing significantly in expert prompt engineers and educational resources rising to meet market demand, designing high-quality prompts has become an intriguing challenge. In this paper, we propose a novel attack against LLMs, named prompt stealing attacks. Our proposed prompt stealing attack aims to steal these well-designed prompts based on the generated answers. The prompt stealing attack contains two primary modules: the parameter extractor and the prompt reconstruction. The goal of the parameter extractor is to figure out the properties of the original prompts. We first observe that most prompts fall into one of three categories: direct prompt, role-based prompt, and in-context prompt. Our parameter extractor first tries to distinguish the type of prompts based on the generated answers. Then, it can further predict which role or how many contexts are used based on the types of prompts. Following the parameter extractor, the prompt reconstructor can be used to reconstruct the original prompts based on the generated answers and the extracted features. The final goal of the prompt reconstructor is to generate the reversed prompts, which are similar to the original prompts. Our experimental results show the remarkable performance of our proposed attacks. Our proposed attacks add a new dimension to the study of prompt engineering and call for more attention to the security issues on LLMs.
Vulnerability of Text-to-Image Models to Prompt Template Stealing: A Differential Evolution Approach
Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；University of California at Merced（加州大学默塞德分校）；University of Chinese Academy of Sciences（中国科学院大学）；The University of Sydney（悉尼大学）
BESA: Boosting Encoder Stealing Attack with Perturbation Recovery
School of Cyberspace Science and Technology, Beijing Institute of Technology（网络安全科学与技术学院，北京理工大学）；Beijing Institute of Technology（北京理工大学）；Singapore University of Technology and Design（新加坡科技设计大学）
Prompt Pirates Need a Map: Stealing Seeds helps Stealing Prompts
Visual Computing and Artificial Intelligence, Kiel University, Germany（视觉计算与人工智能，基尔大学，德国）；Artificial Intelligence, Kiel University, Germany（人工智能，基尔大学，德国）
Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attacks
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）；Saarland Informatics Campus, Germany（德国萨尔州信息学园区）
I Stolenly Swear That I Am Up to (No) Good: Design and Evaluation of Model Stealing Attacks
Christian Doppler Laboratory for Assurance；Transparency in Software Protection, University of Vienna Vienna Austria；SBA Research \& TU Wien Vienna Austria；IBM Research Zurich Zurich Switzerland；Institute of Information Systems Engineering, TU Wien Vienna Austria；Transparency in Software Protection, University of Vienna；SBA Research \& TU Wien；IBM Research Zurich；Institute of Information Systems Engineering, TU Wien
Isolation and Induction: Training Robust Deep Neural Networks against Model Stealing Attacks
Beihang University（北京航空航天大学）；Institute of Dataspace（数据空间研究院）；Chinese Academy of Sciences（中国科学院）；Sensetime Group Limited（商汤集团有限公司）；Zhongguancun Laboratory（中关村实验室）
Medical Multimodal Model Stealing Attacks via Adversarial Domain Alignment
Bosch Center for Artificial Intelligence（博世人工智能中心）；Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Saarland University（萨尔大学）；University of Strasbourg（斯特拉斯堡大学）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）
Link Stealing Attacks Against Inductive Graph Neural Networks
CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全研究中心）；Hong Kong University of Science and Technology（香港科技大学）；University of Birmingham（伯明翰大学）；University of Lausanne（洛桑大学）；Duke University（杜克大学）
Model-Guardian: Protecting against Data-Free Model Stealing Using Gradient Representations and Deceptive Predictions
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；State Key Laboratory of Cyberspace Security Defense（网络空间安全防御国家重点实验室）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
A Model Stealing Attack Against Multi-Exit Networks
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Boston University（波士顿大学）
Graduate School of Computer Science, Saarland University（萨尔大学计算机科学研究生院）；Bosch Center for Artificial Intelligence, Robert Bosch GmbH（罗伯特·博世有限公司博世人工智能中心）；CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）
Towards Effective Prompt Stealing Attack against Text-to-Image Diffusion Models
Nanyang Technological University, Singapore（南洋理工大学）；National University of Singapore, Singapore（新加坡国立大学）；Tsinghua University, China（清华大学）；Agency for Science, Technology and Research (A*STAR), Singapore（科学技术研究局）
