“恐怕我无法做到这一点”：预测黑盒生成式语言模型的提示拒绝行为
I'm Afraid I Can't Do That: Predicting Prompt Refusal in Black-Box Generative Language Models
AI总结：
本研究针对黑盒生成式语言模型的提示拒绝预测问题，通过黑盒攻击构建数据集训练分类器，实现了无需查看模型回复即可预测ChatGPT是否拒绝提示的效果，同时公开了相关数据集与代码。
摘要：
自OpenAI的ChatGPT发布以来，生成式语言模型已引发广泛公众关注。使用量的增长凸显了生成模型的广泛实用性，但也暴露了多种内在偏差。部分偏差由预训练语料库引发；而生成模型特有的另一类偏差则源于为避免生成有害内容而采用的主观微调。微调偏差可能来自工程师个人和公司政策，会影响模型选择拒绝哪些提示。\n在本实验中，我们通过黑盒攻击刻画ChatGPT的拒绝行为。我们首先用各类冒犯性和良性提示（样本量n=1706）查询ChatGPT，随后将每条回复人工标注为合规或拒绝。人工检查回复发现，拒绝并非清晰的二元结果，而是处于连续区间上；因此，我们将多种不同类型的回复映射为合规/拒绝二元标签。我们用这个小型人工标注数据集训练了一个拒绝分类器，其准确率达到96%。\n其次，我们利用该拒绝分类器，基于Quora Insincere Questions数据集构建了一个更大的（样本量n=10000）数据集。借助该机器标注数据，我们训练了一个提示分类器，可在不查看ChatGPT回复的情况下预测其是否会拒绝给定问题。该提示分类器在人工标注的测试集问题（样本量n=985）上达到76%的准确率。我们分析了分类器以及对合规或拒绝最具预测性的提示n元语法。我们的数据集和代码可在https://github.com/maxwellreuter/chatgpt-refusals获取。
英文摘要：
Since the release of OpenAI's ChatGPT, generative language models have attracted extensive public attention. The increased usage has highlighted generative models' broad utility, but also revealed several forms of embedded bias. Some is induced by the pre-training corpus; but additional bias specific to generative models arises from the use of subjective fine-tuning to avoid generating harmful content. Fine-tuning bias may come from individual engineers and company policies, and affects which prompts the model chooses to refuse. In this experiment, we characterize ChatGPT's refusal behavior using a black-box attack. We first query ChatGPT with a variety of offensive and benign prompts (n=1,706), then manually label each response as compliance or refusal. Manual examination of responses reveals that refusal is not cleanly binary, and lies on a continuum; as such, we map several different kinds of responses to a binary of compliance or refusal. The small manually-labeled dataset is used to train a refusal classifier, which achieves an accuracy of 96%. Second, we use this refusal classifier to bootstrap a larger (n=10,000) dataset adapted from the Quora Insincere Questions dataset. With this machine-labeled data, we train a prompt classifier to predict whether ChatGPT will refuse a given question, without seeing ChatGPT's response. This prompt classifier achieves 76% accuracy on a test set of manually labeled questions (n=985). We examine our classifiers and the prompt n-grams that are most predictive of either compliance or refusal. Our datasets and code are available at https://github.com/maxwellreuter/chatgpt-refusals.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
