Polos：面向图像描述的人类反馈多模态度量学习
Polos: Multimodal Metric Learning from Human Feedback for Image Captioning
AI总结：
本文提出 Polos 多模态监督评估指标及 M²LHF 人类反馈训练框架，并构建含 131K 判断的 Polaris 数据集，在多个基准上实现图像描述评估的最优性能。
摘要：
建立一种与人类判断高度一致的自动评估指标，对于有效开发图像描述模型至关重要。近期的数据驱动指标已显示出比 CIDEr 等经典指标更强的人类判断相关性；然而，它们在处理幻觉以及泛化到多样图像和文本方面能力不足，部分原因在于它们仅利用从与图像描述评估无关的任务中学到的嵌入来计算标量相似度。在本研究中，我们提出 Polos，这是一种用于图像描述模型的监督式自动评估指标。Polos 采用并行特征提取机制，利用通过大规模对比学习训练得到的嵌入，从多模态输入计算分数。为训练 Polos，我们引入 Multimodal Metric Learning from Human Feedback（M²LHF），这是一个基于人类反馈开发指标的框架。我们构建了 Polaris 数据集，其中包含来自 550 名评估者的 131K 条人类判断，规模约为标准数据集的十倍。我们的方法在 Composite、Flickr8K-Expert、Flickr8K-CF、PASCAL-50S、FOIL 和 Polaris 数据集上取得了最先进性能，从而证明了其有效性和鲁棒性。
英文摘要：
Establishing an automatic evaluation metric that closely aligns with human judgments is essential for effectively developing image captioning models. Recent data-driven metrics have demonstrated a stronger correlation with human judgments than classic metrics such as CIDEr; however they lack sufficient capabilities to handle hallucinations and generalize across diverse images and texts partially because they compute scalar similarities merely using embeddings learned from tasks unrelated to image captioning evaluation. In this study, we propose Polos, a supervised automatic evaluation metric for image captioning models. Polos computes scores from multimodal inputs, using a parallel feature extraction mechanism that leverages embeddings trained through large-scale contrastive learning. To train Polos, we introduce Multimodal Metric Learning from Human Feedback (M$^2$LHF), a framework for developing metrics based on human feedback. We constructed the Polaris dataset, which comprises 131K human judgments from 550 evaluators, which is approximately ten times larger than standard datasets. Our approach achieved state-of-the-art performance on Composite, Flickr8K-Expert, Flickr8K-CF, PASCAL-50S, FOIL, and the Polaris dataset, thereby demonstrating its effectiveness and robustness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
