基于视觉线索检测手语翻译中的幻觉：是依据视觉信息还是猜测？
Grounding or Guessing? Visual Signals for Detecting Hallucinations in Sign Language Translation
AI总结：
针对手语翻译中模型依赖语言先验而非视觉输入导致幻觉的问题，提出一种基于特征敏感性和反事实信号的令牌级可靠性度量，用于量化视觉信息利用程度，并在两个基准上验证其预测幻觉率、跨数据集泛化及与文本信号结合提升风险评估的效果。
摘要：
幻觉是指模型生成流畅但缺乏视觉证据支持的文本，这是视觉-语言模型的主要缺陷，在手语翻译（SLT）中尤为关键。在SLT中，意义依赖于视频中的精确依据，而无词汇表模型尤其脆弱，因为它们直接将连续的手势运动映射为自然语言，缺少作为对齐的中间词汇表监督。我们认为，当模型依赖语言先验而非视觉输入时会产生幻觉。为捕捉这一点，我们提出一种令牌级可靠性度量，用于量化解码器使用视觉信息的程度。我们的方法结合了基于特征的敏感性（衡量视频被掩蔽时的内部变化）和反事实信号（捕捉干净视频与修改视频之间的概率差异）。这些信号被聚合为句子级可靠性分数，提供了一种紧凑且可解释的视觉依据度量。我们在两个SLT基准（PHOENIX-2014T和CSL-Daily）上，使用基于词汇表和无词汇表模型评估了所提出的度量。结果表明，可靠性能够预测幻觉率，跨数据集和架构泛化，并在视觉退化下降低。除了这些定量趋势，我们还发现可靠性能够区分有依据的令牌和猜测的令牌，从而无需参考即可进行风险估计；当与基于文本的信号（置信度、困惑度或熵）结合时，进一步改善了幻觉风险评估。定性分析揭示了为什么无词汇表模型更容易产生幻觉。综合来看，我们的发现将可靠性确立为诊断SLT中幻觉的实用且可复用的工具，并为多模态生成中更鲁棒的幻觉检测奠定了基础。
英文摘要：
Hallucination, where models generate fluent text unsupported by visual evidence, remains a major flaw in vision-language models and is particularly critical in sign language translation (SLT). In SLT, meaning depends on precise grounding in video, and gloss-free models are especially vulnerable because they map continuous signer movements directly into natural language without intermediate gloss supervision that serves as alignment. We argue that hallucinations arise when models rely on language priors rather than visual input. To capture this, we propose a token-level reliability measure that quantifies how much the decoder uses visual information. Our method combines feature-based sensitivity, which measures internal changes when video is masked, with counterfactual signals, which capture probability differences between clean and altered video inputs. These signals are aggregated into a sentence-level reliability score, providing a compact and interpretable measure of visual grounding. We evaluate the proposed measure on two SLT benchmarks (PHOENIX-2014T and CSL-Daily) with both gloss-based and gloss-free models. Our results show that reliability predicts hallucination rates, generalizes across datasets and architectures, and decreases under visual degradations. Beyond these quantitative trends, we also find that reliability distinguishes grounded tokens from guessed ones, allowing risk estimation without references; when combined with text-based signals (confidence, perplexity, or entropy), it further improves hallucination risk estimation. Qualitative analysis highlights why gloss-free models are more susceptible to hallucinations. Taken together, our findings establish reliability as a practical and reusable tool for diagnosing hallucinations in SLT, and lay the groundwork for more robust hallucination detection in multimodal generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
