扩散式视觉-语言模型中的可靠性挑战
Reliability Challenges in Diffusion Vision-Language Models
浏览论文内容
中文总结 AI 辅助
本文首次系统性评估扩散式视觉-语言模型的可靠性，发现其在二元视觉查询中反转自回归模型的“是”偏差，幻觉率相当但语言质量下降，且存在种族、性别偏差及多项选择任务的准确率崩溃问题。
中文摘要 AI 辅助
基于扩散的大型视觉-语言模型（dLVLMs）近来已成为自回归（AR）式大型视觉-语言模型（LVLMs）的有力替代方案，在并行解码、双向上下文及可控生成方面具备优势。尽管该领域进展迅速，但其可靠性特性仍未得到充分表征。本文首次针对dLVLMs的幻觉与偏差开展系统性可靠性评估，将6个扩散模型与具竞争力的AR基线模型在4个维度上进行基准测试。核心发现包括：（1）在二元视觉查询任务中，dLVLMs反转了AR模型的“是”偏差；（2）其幻觉率与AR模型相当，但语言质量有所下降；（3）在代表性不足的种族群体上，dLVLMs的准确率降至接近零，且存在极性相反的性别偏差；（4）在多项选择任务中，当正确选项比干扰项短时，dLVLMs会出现准确率崩溃，这与首次去噪步骤中出现的长度先验相关。去噪后期步骤中低置信度的 token 与幻觉内容进一步相关，这是扩散生成特有的机制信号。这些模式在不同模型家族间存在差异，表明可靠性由生成范式与训练数据共同塑造。
英文摘要
Diffusion-based Large Vision-Language Models (dLVLMs) have recently emerged as a compelling alternative to autoregressive (AR) LVLMs, offering advantages in parallel decoding, bidirectional context, and controllable generation. Despite rapid progress, their reliability properties remain largely uncharacterized. We present the first systematic reliability evaluation of hallucination and bias in dLVLMs, benchmarking six diffusion models against competitive AR baselines across four dimensions. Our key findings are: (1) dLVLMs reverse the yes-bias of AR models in binary visual queries; (2) they achieve competitive hallucination rates yet exhibit degraded linguistic quality; (3) they collapse to near-zero accuracy on underrepresented racial groups with opposite-polarity gender bias; and (4) they exhibit accuracy collapse in multiple-choice settings when the correct option is shorter than its distractors, associated with a length prior that emerges at the first denoising step. Tokens committed at late denoising steps with low confidence further correlate with hallucinated content, pointing to a mechanistic signal unique to diffusion generation. These patterns vary across model families, suggesting reliability is shaped by the generative paradigm together with training data.
发表机构
- Virginia Tech（弗吉尼亚理工大学）
机构由 AI 辅助整理，请以论文原文为准。
