HKD4VLM：面向视觉语言模型中鲁棒多模态幻觉与事实性检测的渐进式混合知识蒸馏框架
HKD4VLM: A Progressive Hybrid Knowledge Distillation Framework for Robust Multimodal Hallucination and Factuality Detection in VLMs
AI总结：
针对视觉语言模型的幻觉检测与事实性核查任务，提出HKD4VLM渐进式混合知识蒸馏框架，结合分层蒸馏、偏移增强推理与数据增强策略，有效提升检测性能与鲁棒性。
摘要：
在视觉语言模型（VLM）快速发展的推动下，大规模多模态模型的负责任行为已成为重要研究领域，尤其聚焦于幻觉检测与事实性核查。本文提出了针对负责任AI挑战赛两个赛道的解决方案。通用领域的研究启发表明，经过蒸馏的小型VLM在下游任务上的表现往往优于直接微调的大型VLM，同时效率更高。因此，我们从知识蒸馏的角度联合解决两项任务，提出了名为HKD4VLM的渐进式混合知识蒸馏框架。具体而言，该整体框架可分解为金字塔式渐进在线蒸馏与三元耦合精炼蒸馏，分层实现从粗粒度知识对齐到细粒度精炼的过渡。此外，我们进一步引入映射偏移增强推理与多样化增强策略，以提升模型性能与鲁棒性。大量实验结果证明了HKD4VLM的有效性。消融研究揭示了推动性能提升的关键设计选择。
英文摘要：
Driven by the rapid progress in vision-language models (VLMs), the responsible behavior of large-scale multimodal models has become a prominent research area, particularly focusing on hallucination detection and factuality checking. In this paper, we present the solution for the two tracks of Responsible AI challenge. Inspirations from the general domain demonstrate that a smaller distilled VLM can often outperform a larger VLM that is directly tuned on downstream tasks, while achieving higher efficiency. We thus jointly tackle two tasks from the perspective of knowledge distillation and propose a progressive hybrid knowledge distillation framework termed HKD4VLM. Specifically, the overall framework can be decomposed into Pyramid-like Progressive Online Distillation and Ternary-Coupled Refinement Distillation, hierarchically moving from coarse-grained knowledge alignment to fine-grained refinement. Besides, we further introduce the mapping shift-enhanced inference and diverse augmentation strategies to enhance model performance and robustness. Extensive experimental results demonstrate the effectiveness of our HKD4VLM. Ablation studies provide insights into the critical design choices driving performance gains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
