arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.17375cs.CVcs.AI
收藏

当文本劫持视觉:基准测试与减轻文本叠加引起的视觉语言模型幻觉

When Text Hijacks Vision: Benchmarking and Mitigating Text Overlay-Induced Hallucination in Vision Language Models

The Hong Kong University of Science and Technology(香港科学与技术大学);University of Birmingham(伯明翰大学)

Cui Yakun, Xingqun Qi, TianTian Geng, Yuyao Zhang, Sirui Han, Yike Guo

AI总结:

本文提出VisualTextTrap基准测试,通过大规模人工验证样本和定制评估指标,减轻文本叠加引起的幻觉问题,并提出VTHM-MoE框架以提升视觉-文本解耦能力。

摘要:

最近视觉语言模型(VLMs)的进步显著增强了多模态视频理解基准测试中的时间、动作、物体和空间理解能力。然而,我们发现一个关键但被忽视的问题:当屏幕上嵌入的文本与视觉场景矛盾时,现有VLMs系统性地产生幻觉,优先考虑叠加文本语义而非实际视觉内容。我们定义这种现象为文本叠加引起的幻觉(TOIH)。在本文中,我们提出VisualTextTrap,第一个全面的基准测试,包括大规模人工验证样本和专门设计的评估指标。特别是,我们通过VLMs辅助文本生成和严格的人工验证的可扩展混合管道,从广泛使用的公共数据集中构建VisualTextTrap。该基准测试包含6,057个样本,标注于四个维度中的88个细粒度属性,幻觉强度在五级量表(L1--L5)上量化,反映了叠加文本与视觉现实之间的语义矛盾。此外,我们提出Visual Text Hallucination Mitigation Mixture-of-Experts(VTHM-MoE),一种新的视觉-文本解耦框架,采用双编码器架构。具体来说,首先预训练四个维度专用的专家模块,涵盖时间、动作、物体和空间推理,以识别和利用文本语义与实际视频内容之间的跨模态差异。我们开发了自适应令牌路由策略,以实现动态专家分配,从而在抵抗TOIH的同时保持对未受污染视频的性能。在我们的VisualTextTrap基准测试上进行的广泛实验验证了VTHM-MoE的有效性,在多样化的视频问答任务中优于最先进的方法。

英文摘要:

Recent advances in Vision-Language Models (VLMs) have substantially enhanced their ability across multimodal video understanding benchmarks spanning temporal, action, object, and spatial understanding. However, we identify a critical yet overlooked issue: when embedded on-screen text contradicts the visual scene, existing VLMs systematically hallucinate, prioritizing overlay textual semantics over the actual visual content. We define this phenomenon as Text Overlay-Induced Hallucination (TOIH). In this work, we propose VisualTextTrap, the first comprehensive benchmark, including large-scale human-validated samples with specifically designed evaluation metrics. In particular, we construct VisualTextTrap from widely-used public datasets using a scalable hybrid pipeline of VLMs assisted text generation and rigorous manual verification. The benchmark features 6,057 samples annotated across 88 fine-grained attributes within four dimensions, with hallucination intensity quantified on a five-level scale (L1--L5) that reflects the semantic contradiction between overlay text and visual reality. Moreover, we propose Visual Text Hallucination Mitigation Mixture-of-Experts (VTHM-MoE), a novel Vision-Text Disentanglement framework that employs a dual-encoder architecture. Concretely, four dimension-specialized expert modules spanning Temporal, Action, Object, and Spatial reasoning are first pre-trained to identify and leverage cross-modal discrepancies between textual semantics and actual video content. We develop an Adaptive Token Routing Strategy to enable dynamic expert allocation, conferring robust resistance to TOIH while preserving performance on uncontaminated videos. Extensive experiments conducted on our VisualTextTrap benchmark verify the effectiveness of VTHM-MoE, outperforming state-of-the-art counterparts with diverse video question answering tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑