LLM智能体的可靠弱到强监控
Reliable Weak-to-Strong Monitoring of LLM Agents
AI总结：
本文提出监控红队工作流程和混合层次-顺序监控支架，通过SHADE-Arena及新扩展的CUA-SHADE-Arena对LLM智能体隐蔽不当行为进行压力测试，发现智能体感知显著降低监控可靠性，而混合支架可实现弱模型可靠监控强智能体的弱到强扩展效应，并证明针对性人类监督最有效。
摘要：
我们对用于检测自主LLM智能体隐蔽不当行为（例如秘密共享私人信息）的监控系统进行了压力测试。为此，我们系统化了一个监控红队（MRT）工作流程，该流程包含：（1）不同级别的智能体和监控器情境感知；（2）规避监控器的不同对抗策略，例如提示注入；（3）两个数据集和环境——用于工具调用智能体的SHADE-Arena，以及我们新提出的CUA-SHADE-Arena，后者扩展了TheAgentCompany，用于计算机操作智能体。我们在现有的LLM监控支架上运行MRT，这些支架编排LLM并解析智能体轨迹，同时运行本工作提出的新型混合层次-顺序支架。我们的实证结果得出三个关键发现。第一，智能体感知主导监控器感知：智能体知道自己正在被监控会显著降低监控器的可靠性。相反，向监控器提供更多关于智能体的信息不如预期那样有帮助。第二，监控支架比监控器感知更重要：混合支架始终优于基线监控支架，并且可以使较弱的模型可靠地监控较强的智能体——一种弱到强的扩展效应。第三，在人类参与的环境中，人类与LLM监控器讨论以获得对智能体行为的更新判断，针对性的人类监督最为有效；仅将预先标记的案例升级给人类审查员，在FPR = 0.01时将TPR提高了约15%。我们的工作建立了MRT的标准工作流程，突出了LLM和人类在监控和检测智能体不当行为时缺乏对抗鲁棒性的问题。我们发布了代码、数据和日志以促进进一步研究。
英文摘要：
We stress test monitoring systems for detecting covert misbehavior in autonomous LLM agents (e.g., secretly sharing private information). To this end, we systematize a monitor red teaming (MRT) workflow that incorporates: (1) varying levels of agent and monitor situational awareness; (2) distinct adversarial strategies to evade the monitor, such as prompt injection; and (3) two datasets and environments -- SHADE-Arena for tool-calling agents and our new CUA-SHADE-Arena, which extends TheAgentCompany, for computer-use agents. We run MRT on existing LLM monitor scaffoldings, which orchestrate LLMs and parse agent trajectories, alongside a new hybrid hierarchical-sequential scaffolding proposed in this work. Our empirical results yield three key findings. First, agent awareness dominates monitor awareness: an agent's knowledge that it is being monitored substantially degrades the monitor's reliability. On the contrary, providing the monitor with more information about the agent is less helpful than expected. Second, monitor scaffolding matters more than monitor awareness: the hybrid scaffolding consistently outperforms baseline monitor scaffolding, and can enable weaker models to reliably monitor stronger agents -- a weak-to-strong scaling effect. Third, in a human-in-the-loop setting where humans discuss with the LLM monitor to get an updated judgment for the agent's behavior, targeted human oversight is most effective; escalating only pre-flagged cases to human reviewers improved the TPR by approximately 15% at FPR = 0.01. Our work establishes a standard workflow for MRT, highlighting the lack of adversarial robustness for LLMs and humans when monitoring and detecting agent misbehavior. We release code, data, and logs to spur further research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
