UniProbe：基于多结构内部表征的可学习大视觉语言模型（VLM） token 级幻觉检测器
UniProbe: A Learnable Token-Level Hallucination Detector for Large VLMs using Multi-Structural Internal Representations
浏览论文内容
中文总结 AI 辅助
UniProbe 是基于 LVLM 异构计算轨迹的可学习 token 级幻觉检测器，通过多结构模块交互实现 SOTA 检测性能，解码时可降 55% 对象幻觉且延迟仅增 6%
中文摘要 AI 辅助
大视觉语言模型（LVLM）具备出色的视觉推理与对话能力，但常生成与视觉输入不符的幻觉内容。有效缓解该问题需实现 token 级定位，以便针对性干预而无需丢弃整个响应。现有检测器存在不足：需昂贵的全模型微调、依赖忽略模型生成过程的外部验证器、或把内部信号简化为孤立特征与手工统计量，丢失空间、序列与关系结构。本文提出 UniProbe，一种轻量、统一的可学习检测器，通过单次前向传播建模冻结 LVLM 的异构计算轨迹。UniProbe 在图像块、查询 token 与生成 token 间构建带注意力权重（编码其关系）的有向图，通过交替的结构感知模块处理该轨迹：用于关系证据的 GNN、用于二维视觉几何的 ViT、用于响应顺序的 GRU，模块间交织使空间、关系与序列证据在检测器内交互。本文还开发了用于幻觉感知解码的流式变体，可在生成过程中检测并重采样幻觉 token，以及使检测器与 LVLM 自身生成对齐的自适配策略。在多种 LVLM 骨干上，UniProbe 实现了 token 级与对象幻觉检测的 SOTA 性能；解码时，其在延迟为标准生成的 1.06 倍的情况下，将对象幻觉最多降低 55%。
英文摘要
Large Vision-Language Models (LVLMs) achieve impressive visual reasoning and dialogue capabilities, yet frequently hallucinate content unsupported by the visual input. Effective mitigation requires token-level localization, enabling targeted intervention without discarding the entire response. Existing detectors require expensive full-model fine-tuning, rely on external verifiers that ignore the model's generation process, or reduce internal signals to isolated features and hand-crafted statistics, discarding spatial, sequential, and relational structure. We introduce \textbf{UniProbe}, a lightweight, unified, learnable detector that models a frozen LVLM's heterogeneous computational trace from a single forward pass. UniProbe constructs a directed graph over image patches, query tokens, and generated tokens, with attention weights encoding their relations. It processes this trace with alternating structure-aware modules: a GNN for relational evidence, a ViT for 2-D visual geometry, and a GRU for response order. Interleaving them allows spatial, relational, and sequential evidence to interact throughout the detector. We further develop a streaming variant for hallucination-aware decoding, which detects and resamples hallucinated tokens during generation, and a self-adaptation strategy aligning the detector with the LVLM's own generations. Across diverse LVLM backbones, UniProbe achieves state-of-the-art token-level and object-hallucination detection. During decoding, it reduces object hallucinations by up to 55\% at $1.06\times$ the latency of standard generation.
发表机构
- NVIDIA Research（英伟达研究院）
- Technion（以色列理工学院）
- Bar-Ilan University（巴伊兰大学）
- University of Groningen（格罗宁根大学）
机构由 AI 辅助整理，请以论文原文为准。
