We introduce a red-teaming methodology that exposes harder-to-catch attacks for coding-agent monitors, suggesting that current practices may under-elicit attacks and overstate monitor performance. We identify three challenges with current red-teaming. First, mode collapse in attack generation, which we reduce with a novel attack taxonomy for broader coverage. Second, a conceive-execute gap: frontier LLMs can propose strong attack ideas or execute them, but not all at once. We mitigate this by decomposing attack construction into strategy generation, execution, and post-hoc trajectory refinement. Third, manual elicitation is costly to scale, which we address with our semi-automated red-teaming pipeline. Applied to BashArena, an AI control setting for tool-using coding agents, this pipeline produces MonitoringBench, a benchmark of 2,644 attack trajectories for evaluating monitor capabilities and failure modes. Our pipeline produces more diverse and stronger attacks: Opus-4.5 monitor's catch rate falls from 94.9\% on elicited-only Opus attacks to 60.3\% on our best refined attacks, with larger drops for several mid-tier monitors. Attacks optimized against three development monitors generalize to ten held-out monitors, with catch rates generally increasing with monitor capability. Using this benchmark, we provide a snapshot of the current monitor capabilities and find that frontier monitors often detect suspicious actions but fall for persuasion or fail to calibrate suspiciousness scores appropriately, suggesting tractable paths for improvement. MonitoringBench provides both a static benchmark for current tool-use monitors and a reusable methodology for refreshing these evaluations as agents and monitors improve.
Vijil / AI Risk and Vulnerability Alliance(Vijil / AI风险与漏洞联盟);AI Risk and Vulnerability Alliance(AI风险与漏洞联盟);Montreal AI Ethics Institute(蒙特利尔人工智能伦理研究所)
Automatic Red Teaming LLM-based Agents with Model Context Protocol Tools
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Palo Alto Networks(帕洛阿尔托网络公司);School of Software Technology, Zhejiang University(浙江大学软件技术学院)
Whispers of Wealth: Red-Teaming Google's Agent Payments Protocol via Prompt Injection
School of Computing University of Georgia Athens, USA(佐治亚大学计算机学院 美国亚特兰大);Department of Information Technology Kennesaw State University Kennesaw, USA(凯斯韦尔州立大学信息科技学院 美国凯斯韦尔)
Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluation
Responsible Tech Research IBM Research Yorktown Heights New York USA(负责技术研究 IBM 研究 Yorktown Heights 新 York 美国);IBM Research(IBM 研究);The Pennsylvania State University(宾夕法尼亚州立大学);University of Notre Dame(诺特达美大学)
DiffZOO: A Purely Query-Based Black-Box Attack for Red-teaming Text-to-Image Generative Model via Zeroth Order Optimization
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Drexel University(德雷塞尔大学);Shanghai Innovation Center for Processor Technologies, SHIC(上海处理器技术创新中心)
HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Center for AI Safety(人工智能安全中心);Carnegie Mellon University(卡内基梅隆大学);UC Berkeley(加州大学伯克利分校);Microsoft(微软)
Capability-Based Scaling Trends for LLM-Based Red-Teaming
ELLIS Institute Tübingen(图宾根ELLIS研究所);Max Planck Institute for Intelligent Systems(智能系统马克斯·普朗克研究所);Tübingen AI Center(图宾根人工智能中心);Foundation AI – Cisco Systems Inc.(AI基础研究机构——思科系统公司);EPFL(苏黎世联邦理工学院)
Governing Execution Risk in Agentic AI Systems: A Trajectory-Guided Framework for Red Teaming
Department of Information Systems, Business Statistics and Operations Management (ISOM), Hong Kong University of Science and Technology(香港科技大学信息系统、商务统计与运营管理系(ISOM))
DREAM: Scalable Red Teaming for Text-to-Image Generative Systems via Distribution Modeling
School of Cyber Science(网络安全学院);State Key Laboratory of Blockchain and Data Security(区块链与数据安全国家重点实验室);Zhejiang University(浙江大学);Tsinghua University(清华大学)
UDora: A Unified Red Teaming Framework against LLM Agents by Dynamically Hijacking Their Own Reasoning
Department of Computer Science, University of Chicago(芝加哥大学计算机科学系);Meta;Department of Computer Science, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校计算机科学系)