用于评估视觉-语言模型中语气诱导幻觉的LLM-as-Judge框架
LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models
AI总结：
本文提出Ghost-100基准，通过结构化提示强度框架评估不同模型在提示压力下的表现，发现H-Rate和H-Score在模型家族间显著分化，且某些模型对语气敏感度非单调。
摘要：
视觉-语言模型（VLMs）在需要可靠视觉基础的场景中被广泛应用，但其在逐步强制性提示语下的行为仍不明确。现有幻觉基准主要依赖中性提示和二元检测，无法说明在结构上不同的任务类型中，幻觉发生率和强度如何响应分级语言压力。我们提出了Ghost-100，一个由程序生成的基准，包含800张合成图像，涵盖三个任务家族：文本不可读、时间阅读和物体缺失，每个任务都基于负真实原则构建，确保查询目标不存在、不可读或不确定。每张图像配对五个提示，来自结构化的5级提示强度框架，保持图像和任务身份固定，仅改变指令力度，从而将语气作为唯一独立变量。我们采用双轨评估协议：基于规则的H-Rate衡量模型从基于事实的拒绝转向 unsupported 正确承诺的比例，以及由GPT-4o-mini判断的H-Score，用于在发生幻觉时评估其信心和特异性。我们还发布了三阶段自动化验证流程，回顾性确认了800张中的717张严格合规。评估九个开源VLMs，发现H-Rate和H-Score在模型家族间显著分化，阅读风格和存在检测子集对提示压力的响应方式不同，且某些模型对语气敏感度非单调，峰值出现在中间语气水平。
英文摘要：
Vision-Language Models (VLMs) are increasingly deployed in settings where reliable visual grounding carries operational consequences, yet their behavior under progressively coercive prompt phrasing remains undercharacterized. Existing hallucination benchmarks predominantly rely on neutral prompts and binary detection, leaving open how both the incidence and the intensity of fabrication respond to graded linguistic pressure across structurally distinct task types. We present Ghost-100, a procedurally constructed benchmark of 800 synthetically generated images spanning eight categories across three task families: text-illegibility, time-reading, and object-absence, each designed under a negative-ground-truth principle that guarantees the queried target is absent, illegible, or indeterminate by construction. Every image is paired with five prompts drawn from a structured 5-Level Prompt Intensity Framework, holding the image and task identity fixed while varying only directive force, so that tone is isolated as the sole independent variable. We adopt a dual-track evaluation protocol: a rule-based H-Rate measuring the proportion of responses in which a model crosses from grounded refusal into unsupported positive commitment, and a GPT-4o-mini-judged H-Score on a 1-5 scale characterizing the confidence and specificity of fabrication once it occurs. We additionally release a three-stage automated validation workflow, which retrospectively confirms 717 of 800 images as strictly compliant. Evaluating nine open-weight VLMs, we find that H-Rate and H-Score dissociate substantially across model families, reading-style and presence-detection subsets respond to prompt pressure in qualitatively different ways, and several models exhibit non-monotonic sensitivity peaking at intermediate tone levels: patterns that aggregate metrics obscure.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
