Jailbreak Foundry: 从论文到可运行攻击的可重复基准测试
Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking
AI总结：
JAILBREAK FOUNDRY通过多代理工作流将论文转换为可执行模块，实现可重复基准测试的标准化评估和自动化。
摘要：
大型语言模型（LLMs）的逃逸技术发展速度比基准测试快，导致鲁棒性估计过时且难以在不同论文之间比较，因为数据集、利用方法和评估协议会发生漂移。我们介绍了JAILBREAK FOUNDRY（JBF），这是一个通过多代理工作流将逃逸论文转换为可执行模块的系统，以便在统一的利用工具中进行即时评估。JBF有三个核心组件：（i）JBF-LIB用于共享合同和可重用的实用工具；（ii）JBF-FORGE用于多代理论文到模块的转换；（iii）JBF-EVAL用于标准化评估。在30个重现的攻击中，JBF的平均（重现报告）攻击成功率（ASR）偏差为+0.26个百分点。通过利用共享基础设施，JBF将攻击特定的实现代码减少了近一半，相对于原始仓库实现了82.5%的平均重用代码比率。该系统使所有30个攻击在10个受害者模型上使用一致的GPT-4o评估者进行标准化的AdvBench评估。通过自动化攻击集成和标准化评估，JBF提供了一种可扩展的解决方案，用于创建能够跟上快速变化的安全格局的活基准测试。
英文摘要：
Jailbreak techniques for large language models (LLMs) evolve faster than benchmarks, making robustness estimates stale and difficult to compare across papers due to drift in datasets, harnesses, and judging protocols. We introduce JAILBREAK FOUNDRY (JBF), a system that addresses this gap via a multi-agent workflow to translate jailbreak papers into executable modules for immediate evaluation within a unified harness. JBF features three core components: (i) JBF-LIB for shared contracts and reusable utilities; (ii) JBF-FORGE for the multi-agent paper-to-module translation; and (iii) JBF-EVAL for standardizing evaluations. Across 30 reproduced attacks, JBF achieves high fidelity with a mean (reproduced-reported) attack success rate (ASR) deviation of +0.26 percentage points. By leveraging shared infrastructure, JBF reduces attack-specific implementation code by more than half relative to original repositories and achieves an 82.5% mean reused-code ratio. This system enables a standardized AdvBench evaluation of all 30 attacks across 10 victim models using a consistent GPT-4o judge. By automating both attack integration and standardized evaluation, JBF offers a scalable solution for creating living benchmarks that keep pace with the rapidly shifting security landscape.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
