重新思考开放词汇分割的评估指标
Rethinking Evaluation Metrics of Open-Vocabulary Segmentaion
AI总结：
针对开放词汇分割现有闭集评估指标未考虑类别相似性的问题，调研11种类别词相似性度量，设计适配三类任务的Open mIoU等新指标，经12种方法基准测试验证可有效评估模型开放能力。
摘要：
本文指出了开放词汇分割所采用的评估指标存在的一个问题：在零样本或跨数据集流程中，评估过程仍严重依赖闭集指标，未考虑预测类别与真实类别之间的相似性。为解决该问题，我们首先利用WordNet语言学统计数据、文本嵌入和语言模型，通过全面的定量分析与用户研究，调研了两个类别词之间的11种相似性度量方法。基于这些探究得到的度量方式，我们设计了分别适配三类开放词汇分割任务的新型评估指标，即Open mIoU、Open AP和Open PQ。我们在三类分割任务的12种开放词汇方法上对所提出的评估指标进行了基准测试。尽管相似性距离存在相对主观性，我们证明了所提指标仍能很好地评估现有开放词汇分割方法的开放能力。我们希望本工作能为学界带来关于如何评估模型开放能力的新思考，评估代码已在GitHub发布。
英文摘要：
In this paper, we highlight a problem of evaluation metrics adopted in the open-vocabulary segmentation. That is, the evaluation process still heavily relies on closed-set metrics on zero-shot or cross-dataset pipelines without considering the similarity between predicted and ground truth categories. To tackle this issue, we first survey eleven similarity measurements between two categorical words using WordNet linguistics statistics, text embedding, and language models by comprehensive quantitative analysis and user study. Built upon those explored measurements, we designed novel evaluation metrics, namely Open mIoU, Open AP, and Open PQ, tailored for three open-vocabulary segmentation tasks. We benchmarked the proposed evaluation metrics on 12 open-vocabulary methods of three segmentation tasks. Even though the relative subjectivity of similarity distance, we demonstrate that our metrics can still well evaluate the open ability of the existing open-vocabulary segmentation methods. We hope that our work can bring with the community new thinking about how to evaluate the open ability of models. The evaluation code is released in github.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
