HICEScore：一种用于图像描述评估的层次化指标
HICEScore: A Hierarchical Metric for Image Captioning Evaluation
AI总结：
提出无参考的图像描述评估指标 HICE-S，通过层次化评分机制检测局部视觉区域和文本短语，解决现有指标难以评估细节描述和缺乏可解释性的问题，在多个基准上达到 SOTA。
摘要：
图像描述评估指标可分为两类：基于参考的指标和无参考指标。然而，基于参考的方法由于严重依赖有限的人工标注参考，可能难以评估由先进多模态大语言模型生成的包含丰富视觉细节的描述性文本。相比之下，以往的无参考指标通过 CLIP 跨模态相似度已被证明有效。尽管如此，基于 CLIP 的指标受限于其全局图文兼容性的解决方案，通常在检测局部文本幻觉方面存在缺陷，且对微小视觉对象不敏感。此外，其单尺度设计无法提供可解释的评估过程，例如精确定位描述错误的位置以及识别未被描述的视觉区域。为了推进这一领域，我们提出了一种新的无参考图像描述评估指标，称为层次化图像描述评估得分（HICE-S）。通过检测局部视觉区域和文本短语，HICE-S 构建了可解释的层次化评分机制，突破了现有无参考指标单尺度结构的限制。综合实验表明，我们提出的指标在多个基准上取得了 SOTA 性能，优于 CLIP-S 和 PAC-S 等现有无参考指标，以及 METEOR 和 CIDEr 等基于参考的指标。此外，多个案例研究表明，HICE-S 对详细描述的评估过程与可解释的人类判断高度相似。代码可在 https://github.com/joeyz0z/HICE 获取。
英文摘要：
Image captioning evaluation metrics can be divided into two categories, reference-based metrics and reference-free metrics. However, reference-based approaches may struggle to evaluate descriptive captions with abundant visual details produced by advanced multimodal large language models, due to their heavy reliance on limited human-annotated references. In contrast, previous reference-free metrics have been proven effective via CLIP cross-modality similarity. Nonetheless, CLIP-based metrics, constrained by their solution of global image-text compatibility, often have a deficiency in detecting local textual hallucinations and are insensitive to small visual objects. Besides, their single-scale designs are unable to provide an interpretable evaluation process such as pinpointing the position of caption mistakes and identifying visual regions that have not been described. To move forward, we propose a novel reference-free metric for image captioning evaluation, dubbed Hierarchical Image Captioning Evaluation Score (HICE-S). By detecting local visual regions and textual phrases, HICE-S builds an interpretable hierarchical scoring mechanism, breaking through the barriers of the single-scale structure of existing reference-free metrics. Comprehensive experiments indicate that our proposed metric achieves the SOTA performance on several benchmarks, outperforming existing reference-free metrics like CLIP-S and PAC-S, and reference-based metrics like METEOR and CIDEr. Moreover, several case studies reveal that the assessment process of HICE-S on detailed captions closely resembles interpretable human judgments.Our code is available at https://github.com/joeyz0z/HICE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
