关于3D大视觉-语言模型的对抗鲁棒性
On the Adversarial Robustness of 3D Large Vision-Language Models
AI总结：
本文研究了3D大视觉-语言模型的对抗鲁棒性，提出两种攻击策略评估其鲁棒性，发现其在无目标攻击下较脆弱，但在有目标攻击中更稳健。
摘要：
3D视觉-语言模型（VLMs），如PointLLM和GPT4Point，在3D理解任务中展现了强大的推理和泛化能力。然而，其对抗鲁棒性仍 largely 未被探索。先前2D VLMs 的研究显示，视觉输入的整合显著增加了对对抗攻击的脆弱性，使这些模型更容易被操控以生成有毒或误导性的输出。在本文中，我们探讨是否将3D视觉整合会削弱3D VLMs的鲁棒性。为此，我们首次系统研究了基于点的3D VLMs的对抗鲁棒性。我们提出了两种互补的攻击策略：Vision Attack，通过扰动3D编码器和投影器产生的视觉token特征来评估视觉-语言对齐的鲁棒性；以及Caption Attack，直接操控输出token序列以评估端到端系统的鲁棒性。每种攻击都包含无目标和有目标变体，以测量通用脆弱性和对受控操控的易受性。我们的实验发现，3D VLMs在无目标攻击下表现出显著的对抗脆弱性，而在针对强制特定有害输出的有目标攻击中，其鲁棒性比2D模型更强。这些发现突显了改进3D VLMs对抗鲁棒性的重要性，尤其是在它们应用于安全关键应用时。
英文摘要：
3D Vision-Language Models (VLMs), such as PointLLM and GPT4Point, have shown strong reasoning and generalization abilities in 3D understanding tasks. However, their adversarial robustness remains largely unexplored. Prior work in 2D VLMs has shown that the integration of visual inputs significantly increases vulnerability to adversarial attacks, making these models easier to manipulate into generating toxic or misleading outputs. In this paper, we investigate whether incorporating 3D vision similarly compromises the robustness of 3D VLMs. To this end, we present the first systematic study of adversarial robustness in point-based 3D VLMs. We propose two complementary attack strategies: \textit{Vision Attack}, which perturbs the visual token features produced by the 3D encoder and projector to assess the robustness of vision-language alignment; and \textit{Caption Attack}, which directly manipulates output token sequences to evaluate end-to-end system robustness. Each attack includes both untargeted and targeted variants to measure general vulnerability and susceptibility to controlled manipulation. Our experiments reveal that 3D VLMs exhibit significant adversarial vulnerabilities under untargeted attacks, while demonstrating greater resilience against targeted attacks aimed at forcing specific harmful outputs, compared to their 2D counterparts. These findings highlight the importance of improving the adversarial robustness of 3D VLMs, especially as they are deployed in safety-critical applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
