为什么这么毒？测量和触发开放域聊天机器人中的有毒行为
Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbots
摘要：
聊天机器人用于许多应用中，例如，自动代理、灵巧家庭助理、在线游戏中的交互角色等。因此，确保它们不会以不受欢迎的方式表现，从而向用户提供冒犯性或有害的响应，这一点至关重要。这不是一个简单的任务，因为最先进的聊天机器人模型是在从互联网上公开收集的大型公共数据集上训练的。本文首次对聊天机器人的毒性进行了大规模测量。我们发现，公开可用的聊天机器人在提供有毒查询时容易提供有毒响应。更令人担忧的是，一些无毒查询也可能引发有毒响应。然后，我们开始设计和实验一种攻击，ToxicBuddy，它依赖于微调GPT-2来生成无毒查询，使聊天机器人以有毒的方式响应。我们广泛的实验评估表明，我们的攻击对公共聊天机器人模型是有效的，并优于以前的工作提出的手动制作的恶意查询。我们还评估了针对ToxicBuddy的三种防御机制，表明它们要么以影响聊天机器人的效用为代价降低攻击性能，要么只能有效减轻一部分攻击。这凸显了计算机安全和在线安全社区需要进行更多研究，以确保聊天机器人模型不会伤害其用户。总的来说，我们相信ToxicBuddy可以用作审计工具，我们的工作将为设计更有效的聊天机器人安全防御铺平道路。
英文摘要：
Chatbots are used in many applications, e.g., automated agents, smart home assistants, interactive characters in online games, etc. Therefore, it is crucial to ensure they do not behave in undesired manners, providing offensive or toxic responses to users. This is not a trivial task as state-of-the-art chatbot models are trained on large, public datasets openly collected from the Internet. This paper presents a first-of-its-kind, large-scale measurement of toxicity in chatbots. We show that publicly available chatbots are prone to providing toxic responses when fed toxic queries. Even more worryingly, some non-toxic queries can trigger toxic responses too. We then set out to design and experiment with an attack, ToxicBuddy, which relies on fine-tuning GPT-2 to generate non-toxic queries that make chatbots respond in a toxic manner. Our extensive experimental evaluation demonstrates that our attack is effective against public chatbot models and outperforms manually-crafted malicious queries proposed by previous work. We also evaluate three defense mechanisms against ToxicBuddy, showing that they either reduce the attack performance at the cost of affecting the chatbot's utility or are only effective at mitigating a portion of the attack. This highlights the need for more research from the computer security and online safety communities to ensure that chatbot models do not hurt their users. Overall, we are confident that ToxicBuddy can be used as an auditing tool and that our work will pave the way toward designing more effective defenses for chatbot safety.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
