大多模态模型时代下的指代表达理解评估再审视
Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Models
AI总结：
本研究针对现有REC基准标注错误率高、无法全面评估大多模态模型能力的问题，清理了三类经典基准并推出含4.5万余标注的Ref-L4新基准，评估了24个大模型并提供了相关洞见与开源资源。
摘要：
指代表达理解（REC）旨在根据文本描述定位目标实例。近期REC领域的进展由CogVLM等大多模态模型（LMMs）推动，这类模型在RefCOCO上达到了92.44%的准确率。然而，本研究对RefCOCO、RefCOCO+和RefCOCOg等现有基准是否能全面衡量LMMs的能力提出了质疑。我们首先对这些基准进行了人工核查，发现其标注错误率较高：RefCOCO为14%，RefCOCO+为24%，RefCOCOg为5%，这削弱了评估结果的真实性。我们通过排除有问题的样本解决该问题，并重新评估了多个可执行REC任务的LMMs，结果显示准确率有显著提升，由此凸显了基准噪声的影响。针对这一问题，我们推出了Ref-L4这一综合性REC基准，专门用于评估现代REC模型。Ref-L4具备四大核心特征：1）样本规模庞大，包含45341条标注；2）物体类别多样，涵盖365种不同类型，实例数量规模从30到3767不等；3）指代表达篇幅较长，平均长度为24.2个单词；4）词汇量丰富，包含22813个唯一词。我们在Ref-L4上共评估了24个大模型，并提供了有价值的洞见。RefCOCO、RefCOCO+和RefCOCOg的清理版本，以及我们的Ref-L4基准和评估代码，均可在https://github.com/JierunChen/Ref-L4获取。
英文摘要：
Referring expression comprehension (REC) involves localizing a target instance based on a textual description. Recent advancements in REC have been driven by large multimodal models (LMMs) like CogVLM, which achieved 92.44% accuracy on RefCOCO. However, this study questions whether existing benchmarks such as RefCOCO, RefCOCO+, and RefCOCOg, capture LMMs' comprehensive capabilities. We begin with a manual examination of these benchmarks, revealing high labeling error rates: 14% in RefCOCO, 24% in RefCOCO+, and 5% in RefCOCOg, which undermines the authenticity of evaluations. We address this by excluding problematic instances and reevaluating several LMMs capable of handling the REC task, showing significant accuracy improvements, thus highlighting the impact of benchmark noise. In response, we introduce Ref-L4, a comprehensive REC benchmark, specifically designed to evaluate modern REC models. Ref-L4 is distinguished by four key features: 1) a substantial sample size with 45,341 annotations; 2) a diverse range of object categories with 365 distinct types and varying instance scales from 30 to 3,767; 3) lengthy referring expressions averaging 24.2 words; and 4) an extensive vocabulary comprising 22,813 unique words. We evaluate a total of 24 large models on Ref-L4 and provide valuable insights. The cleaned versions of RefCOCO, RefCOCO+, and RefCOCOg, as well as our Ref-L4 benchmark and evaluation code, are available at https://github.com/JierunChen/Ref-L4.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
