从描述丰富性到偏差：揭示生成式图像字幕增强的阴暗面
From Descriptive Richness to Bias: Unveiling the Dark Side of Generative Image Caption Enrichment
AI总结：
该研究揭示生成式图像字幕增强（GCE）虽能提升字幕描述性与视觉对齐度，但会加剧性别偏差与幻觉问题，基于其训练的模型偏差平均放大30.9%、幻觉增加59.5%，为相关趋势敲响警钟。
摘要：
大语言模型（LLMs）提升了视觉语言模型为视觉内容生成字幕的能力。这种生成式图像字幕增强方法进一步让文本字幕更具描述性，改善了与视觉语境的对齐效果。然而，尽管许多研究聚焦于生成式字幕增强（GCE）的益处，它是否存在负面副作用呢？我们从“性别偏差”和“幻觉”两个角度对比了标准格式字幕与近期的GCE流程，发现增强后的字幕存在性别偏差加剧、幻觉增多的问题。此外，基于这些增强字幕训练的模型，性别偏差平均放大30.9%，幻觉现象增加59.5%。本研究对追求字幕描述性增强的趋势提出了警示。
英文摘要：
Large language models (LLMs) have enhanced the capacity of vision-language models to caption visual text. This generative approach to image caption enrichment further makes textual captions more descriptive, improving alignment with the visual context. However, while many studies focus on benefits of generative caption enrichment (GCE), are there any negative side effects? We compare standard-format captions and recent GCE processes from the perspectives of "gender bias" and "hallucination", showing that enriched captions suffer from increased gender bias and hallucination. Furthermore, models trained on these enriched captions amplify gender bias by an average of 30.9% and increase hallucination by 59.5%. This study serves as a caution against the trend of making captions more descriptive.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
