arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.01592cs.CRcs.CV
收藏

OpenRT: 一种用于多模态大语言模型的开源红队框架

OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs

Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)

Xin Wang, Yunhao Chen, Juncheng Li, Yixu Wang, Yang Yao, Tianle Gu, Jie Li, Yan Teng, Yingchun Wang, Xia Hu

AI总结:

OpenRT框架通过模块化设计和高吞吐量运行时,系统性评估多模态大语言模型的安全性,揭示了前沿模型在复杂攻击下的脆弱性。

摘要:

多模态大语言模型(MLLMs)的快速整合到关键应用中,日益受到持续存在的安全漏洞的阻碍。然而,现有的红队基准测试通常碎片化,局限于单轮文本交互,并缺乏系统评估所需的可扩展性。为此,我们介绍了OpenRT,一种统一的、模块化和高吞吐量的红队框架,用于全面的MLLM安全评估。在核心,OpenRT通过引入一个对抗内核,实现了自动化红队方法的范式转变,使五个关键维度实现模块化分离:模型集成、数据集管理、攻击策略、判断方法和评估指标。通过标准化攻击接口,它将对抗逻辑与高吞吐量的异步运行时解耦,使在不同模型上实现系统扩展成为可能。我们的框架集成了37种多样的攻击方法,涵盖白盒梯度、多模态扰动和复杂的多代理进化策略。通过在20种先进模型(包括GPT-5.2、Claude 4.5和Gemini 3 Pro)上的广泛实证研究,我们揭示了关键的安全漏洞:即使前沿模型也无法在不同攻击范式中泛化,领先模型的平均攻击成功率高达49.14%。值得注意的是,我们的发现表明,推理模型并不 inherently 具有更强的抗复杂、多轮 jailbreak 的鲁棒性。通过开源OpenRT,我们提供了一个可持续、可扩展且持续维护的基础设施,加速了AI安全的发展和标准化。

英文摘要:

The rapid integration of Multimodal Large Language Models (MLLMs) into critical applications is increasingly hindered by persistent safety vulnerabilities. However, existing red-teaming benchmarks are often fragmented, limited to single-turn text interactions, and lack the scalability required for systematic evaluation. To address this, we introduce OpenRT, a unified, modular, and high-throughput red-teaming framework designed for comprehensive MLLM safety evaluation. At its core, OpenRT architects a paradigm shift in automated red-teaming by introducing an adversarial kernel that enables modular separation across five critical dimensions: model integration, dataset management, attack strategies, judging methods, and evaluation metrics. By standardizing attack interfaces, it decouples adversarial logic from a high-throughput asynchronous runtime, enabling systematic scaling across diverse models. Our framework integrates 37 diverse attack methodologies, spanning white-box gradients, multi-modal perturbations, and sophisticated multi-agent evolutionary strategies. Through an extensive empirical study on 20 advanced models (including GPT-5.2, Claude 4.5, and Gemini 3 Pro), we expose critical safety gaps: even frontier models fail to generalize across attack paradigms, with leading models exhibiting average Attack Success Rates as high as 49.14%. Notably, our findings reveal that reasoning models do not inherently possess superior robustness against complex, multi-turn jailbreaks. By open-sourcing OpenRT, we provide a sustainable, extensible, and continuously maintained infrastructure that accelerates the development and standardization of AI safety.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑