策略合规智能体的有效红队测试
Effective Red-Teaming of Policy-Adherent Agents
AI总结：
针对基于LLM的策略合规智能体易受恶意用户利用的问题，提出聚焦牟利对抗用户的威胁模型与多智能体红队测试系统CRAFT，引入tau-break基准评估鲁棒性，评估防御策略并指出需更强保护措施。
摘要：
面向任务的基于LLM的智能体正越来越多地应用于具有严格策略的领域，如退款资格或取消规则。挑战在于确保智能体始终遵守这些规则和策略，适当地拒绝任何违反规则的请求，同时仍保持有帮助且自然的交互。这需要开发定制的设计和评估方法，以确保智能体对恶意用户行为具有抵御能力。我们提出了一种新颖的威胁模型，重点关注旨在利用策略合规智能体谋取个人利益的对抗性用户。为解决此问题，我们提出了CRAFT——一种多智能体红队测试系统，该系统利用策略感知的说服策略在客户服务场景中破坏策略合规智能体，其性能优于DAN提示、情感操纵和胁迫等传统越狱方法。基于现有的tau-bench基准，我们引入了tau-break——一个互补基准，旨在严格评估智能体对操纵性用户行为的鲁棒性。最后，我们评估了几种简单但有效的防御策略。尽管这些措施提供了一定的保护，但仍有不足，这凸显了需要更强的、以研究为导向的保护措施来保护策略合规智能体免受对抗性攻击。
英文摘要：
Task-oriented LLM-based agents are increasingly used in domains with strict policies, such as refund eligibility or cancellation rules. The challenge lies in ensuring that the agent consistently adheres to these rules and policies, appropriately refusing any request that would violate them, while still maintaining a helpful and natural interaction. This calls for the development of tailored design and evaluation methodologies to ensure agent resilience against malicious user behavior. We propose a novel threat model that focuses on adversarial users aiming to exploit policy-adherent agents for personal benefit. To address this, we present CRAFT, a multi-agent red-teaming system that leverages policy-aware persuasive strategies to undermine a policy-adherent agent in a customer-service scenario, outperforming conventional jailbreak methods such as DAN prompts, emotional manipulation, and coercive. Building upon the existing tau-bench benchmark, we introduce tau-break, a complementary benchmark designed to rigorously assess the agent's robustness against manipulative user behavior. Finally, we evaluate several straightforward yet effective defense strategies. While these measures provide some protection, they fall short, highlighting the need for stronger, research-driven safeguards to protect policy-adherent agents from adversarial attacks
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
