VisualLeakBench: 对大型视觉-语言模型在PII泄露和社会工程中的脆弱性进行审计
VisualLeakBench: Auditing the Fragility of Large Vision-Language Models against PII Leakage and Social Engineering
AI总结：
本文提出VisualLeakBench，通过合成对抗图像评估LVLMs对OCR注入和PII泄露的鲁棒性，揭示了Claude~4在PII识别上的高风险及防御策略的模板依赖性。
摘要：
随着大型视觉-语言模型（LVLMs）在代理集成工作流等场景中的广泛应用，其对语义视觉攻击的鲁棒性评估不足——对齐通常测试显式有害内容而非隐私关键的多模态场景。我们引入VisualLeakBench，一个评估套件，利用1000张合成对抗图像（包含8种PII类型）和50张真实世界截图（IRL）来审计LVLMs对OCR注入和上下文PII泄露的防御能力。我们评估了四个前沿系统（GPT-5.2、Claude~4、Gemini-3 Flash、Grok-4）并采用Wilson 95%置信区间。Claude~4在OCR ASR上最低（14.2%）但PII ASR最高（74.4%），表现出一种先披露数据再警示的模式。Grok-4在PII ASR上最低（20.4%）。防御系统提示可消除两种模型的PII泄露，将Claude~4的泄露率从74.4%降至2.2%，但对Gemini-3 Flash无影响。令人惊讶的是，IRL验证显示Gemini-3 Flash在真实世界图像上对缓解措施有响应（50%到0%），表明缓解鲁棒性取决于模板而非普遍缺失。我们发布数据集和代码以实现部署相关视觉-语言系统的可重复鲁棒性和安全评估。
英文摘要：
As Large Vision-Language Models (LVLMs) are increasingly deployed in agent-integrated workflows and other deployment-relevant settings, their robustness against semantic visual attacks remains under-evaluated -- alignment is typically tested on explicit harmful content rather than privacy-critical multimodal scenarios. We introduce VisualLeakBench, an evaluation suite to audit LVLMs against OCR Injection and Contextual PII Leakage using 1,000 synthetically generated adversarial images with 8 PII types, validated on 50 in-the-wild (IRL) real-world screenshots spanning diverse visual contexts. We evaluate four frontier systems (GPT-5.2, Claude~4, Gemini-3 Flash, Grok-4) with Wilson 95% confidence intervals. Claude~4 achieves the lowest OCR ASR (14.2%) but the highest PII ASR (74.4%), exhibiting a comply-then-warn pattern -- where verbatim data disclosure precedes any safety-oriented language. Grok-4 achieves the lowest PII ASR (20.4%). A defensive system prompt eliminates PII leakage for two models, reduces Claude~4's leakage from 74.4% to 2.2%, but has no effect on Gemini-3 Flash on synthetic data. Strikingly, IRL validation reveals Gemini-3 Flash does respond to mitigation on real-world images (50% to 0%), indicating that mitigation robustness is template-sensitive rather than uniformly absent. We release our dataset and code for reproducible robustness and safety evaluation of deployment-relevant vision-language systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
