Hydra：一种用于增强视觉-语言模型对抗鲁棒性并缓解幻觉的智能体推理方法
Hydra: An Agentic Reasoning Approach for Enhancing Adversarial Robustness and Mitigating Hallucinations in Vision-Language Models
AI总结：
Hydra提出一种自适应智能体推理框架，通过迭代推理、批评与跨模型验证，同时增强视觉-语言模型的对抗鲁棒性和缓解幻觉，无需训练即可超越现有方法。
摘要：
为了开发可信赖的视觉-语言模型（VLM），必须同时解决对抗鲁棒性和幻觉缓解问题，这两者都会影响国防和医疗保健等高危应用中的事实准确性。现有方法主要侧重于对抗防御或幻觉事后纠正，缺乏统一的鲁棒性策略。我们提出了Hydra，一种自适应智能体框架，通过迭代推理、结构化批评和跨模型验证来增强即插式VLM，同时提高对对抗扰动和内在模型错误的鲁棒性。Hydra采用行动-批评循环，检索并批评视觉信息，利用思维链（CoT）和上下文学习（ICL）技术动态优化输出。与静态事后纠正方法不同，Hydra能适应对抗操纵和内在模型错误，从而对恶意扰动和幻觉相关不准确性具有鲁棒性。我们在四个VLM、三个幻觉基准、两种对抗攻击策略和两种对抗防御方法上评估了Hydra，在干净输入和对抗输入上均评估了性能。结果表明，即使没有显式对抗防御，Hydra也超越了即插式VLM和最新（SOTA）去幻觉方法，展示了增强的鲁棒性和事实一致性。通过桥接对抗抵抗和幻觉缓解，Hydra提供了一种可扩展、无需训练的解决方案，以提高VLM在现实应用中的可靠性。
英文摘要：
To develop trustworthy Vision-Language Models (VLMs), it is essential to address adversarial robustness and hallucination mitigation, both of which impact factual accuracy in high-stakes applications such as defense and healthcare. Existing methods primarily focus on either adversarial defense or hallucination post-hoc correction, leaving a gap in unified robustness strategies. We introduce \textbf{Hydra}, an adaptive agentic framework that enhances plug-in VLMs through iterative reasoning, structured critiques, and cross-model verification, improving both resilience to adversarial perturbations and intrinsic model errors. Hydra employs an Action-Critique Loop, where it retrieves and critiques visual information, leveraging Chain-of-Thought (CoT) and In-Context Learning (ICL) techniques to refine outputs dynamically. Unlike static post-hoc correction methods, Hydra adapts to both adversarial manipulations and intrinsic model errors, making it robust to malicious perturbations and hallucination-related inaccuracies. We evaluate Hydra on four VLMs, three hallucination benchmarks, two adversarial attack strategies, and two adversarial defense methods, assessing performance on both clean and adversarial inputs. Results show that Hydra surpasses plug-in VLMs and state-of-the-art (SOTA) dehallucination methods, even without explicit adversarial defenses, demonstrating enhanced robustness and factual consistency. By bridging adversarial resistance and hallucination mitigation, Hydra provides a scalable, training-free solution for improving the reliability of VLMs in real-world applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
