The rapid integration of Multimodal Large Language Models (MLLMs) into critical applications is increasingly hindered by persistent safety vulnerabilities. However, existing red-teaming benchmarks are often fragmented, limited to single-turn text interactions, and lack the scalability required for systematic evaluation. To address this, we introduce OpenRT, a unified, modular, and high-throughput red-teaming framework designed for comprehensive MLLM safety evaluation. At its core, OpenRT architects a paradigm shift in automated red-teaming by introducing an adversarial kernel that enables modular separation across five critical dimensions: model integration, dataset management, attack strategies, judging methods, and evaluation metrics. By standardizing attack interfaces, it decouples adversarial logic from a high-throughput asynchronous runtime, enabling systematic scaling across diverse models. Our framework integrates 37 diverse attack methodologies, spanning white-box gradients, multi-modal perturbations, and sophisticated multi-agent evolutionary strategies. Through an extensive empirical study on 20 advanced models (including GPT-5.2, Claude 4.5, and Gemini 3 Pro), we expose critical safety gaps: even frontier models fail to generalize across attack paradigms, with leading models exhibiting average Attack Success Rates as high as 49.14%. Notably, our findings reveal that reasoning models do not inherently possess superior robustness against complex, multi-turn jailbreaks. By open-sourcing OpenRT, we provide a sustainable, extensible, and continuously maintained infrastructure that accelerates the development and standardization of AI safety.
Vijil / AI Risk and Vulnerability Alliance(Vijil / AI风险与漏洞联盟);AI Risk and Vulnerability Alliance(AI风险与漏洞联盟);Montreal AI Ethics Institute(蒙特利尔人工智能伦理研究所)
DiffZOO: A Purely Query-Based Black-Box Attack for Red-teaming Text-to-Image Generative Model via Zeroth Order Optimization
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Drexel University(德雷塞尔大学);Shanghai Innovation Center for Processor Technologies, SHIC(上海处理器技术创新中心)
Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
Summon a Demon and Bind it: A Grounded Theory of LLM Red Teaming
Paul G. Allen School of Computer Science & Engineering(保罗·G·艾伦计算机科学与工程学院);IT University of Copenhagen(哥本哈根信息技术大学);Berkeley Center for Human-Compatible AI, UC Berkeley(加州大学伯克利分校伯克利人类兼容人工智能中心);NVIDIA Corporation(英伟达公司)
Automatic Red Teaming LLM-based Agents with Model Context Protocol Tools
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Palo Alto Networks(帕洛阿尔托网络公司);School of Software Technology, Zhejiang University(浙江大学软件技术学院)
DREAM: Scalable Red Teaming for Text-to-Image Generative Systems via Distribution Modeling
School of Cyber Science(网络安全学院);State Key Laboratory of Blockchain and Data Security(区块链与数据安全国家重点实验室);Zhejiang University(浙江大学);Tsinghua University(清华大学)
Learning diverse attacks on large language models for robust red-teaming and safety tuning
KAIST(韩国科学技术院);Mila – Québec AI Institute(米拉-魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);McGill University(麦吉尔大学);National University of Singapore(新加坡国立大学);University of Edinburgh(爱丁堡大学);CIFAR(加拿大高级研究所)
Resource Consumption Red-Teaming for Large Vision-Language Models
China Mobile Research Institute(中国移动研究院);Beijing University of Posts and Telecommunications(北京邮电大学);Nanyang Technological University(南洋理工大学);University of Science and Technology of China(中国科学技术大学);North China Electric Power University(华北电力大学)
Prompt Optimization and Evaluation for LLM Automated Red Teaming
Fuel iX Applied Research(Fuel iX应用研究);North Carolina State University(北卡罗来纳州立大学);University of Minnesota(明尼苏达大学);TELUS Digital(TELUS数字);Learn Prompting
RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent
The State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全国家重点实验室);School of Cyber Science and Technology, Zhejiang University(浙江大学网络科学与技术学院);Palo Alto Networks(派拓网络);University of North Texas(北得克萨斯大学)
RedEdit: Agentic Red-Teaming of Image Safety Classifiers via MCTS-Guided Photo-Editing
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Weixin AI, Tencent Inc.(腾讯微信AI);School of Science, Sun Yat-sen University(中山大学理学院)
Capability-Based Scaling Trends for LLM-Based Red-Teaming
ELLIS Institute Tübingen(图宾根ELLIS研究所);Max Planck Institute for Intelligent Systems(智能系统马克斯·普朗克研究所);Tübingen AI Center(图宾根人工智能中心);Foundation AI – Cisco Systems Inc.(AI基础研究机构——思科系统公司);EPFL(苏黎世联邦理工学院)
Red-Teaming Text-to-Image Systems by Rule-based Preference Modeling
KLMM, UCAS, Academy of Mathematics and Systems Science, Chinese Academy of Sciences(UCAS信息科学学院、数学与系统科学研究院、中国科学院);College of AI, Tsinghua University(清华大学人工智能学院)
MAStrike: Shapley-Guided Collusive Red-Teaming on Multi-Agent Systems
University of Illinois, Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Virtue AI;University of Chicago(芝加哥大学);Wells Fargo(摩根大通);Salesforce;University of California, Santa Barbara(加州大学圣芭芭拉分校)