PinPoint：评估组合图像检索的综合基准，包含显式负样本、多图像查询和同义词测试
PinPoint: Evaluation of Composed Image Retrieval with Explicit Negatives, Multi-Image Queries, and Paraphrase Testing
AI总结：
PinPoint提出一个综合基准测试，通过多正确答案、显式负样本和同义词测试评估组合图像检索的鲁棒性和公平性。
摘要：
组合图像检索（CIR）已取得显著进展，但当前基准测试仅限于单个真实答案，缺乏评估假阳性避免、鲁棒性和多图像推理所需注释。我们提出了PinPoint，一个包含7635个查询和329,000个相关性判断的综合现实世界基准测试，覆盖23个查询类别。PinPoint通过提供：（1）多个正确答案（平均每个查询9.1个）（2）显式困难负样本（3）每个查询六个指令同义词以进行鲁棒性测试（4）多图像组合支持（13.4%的查询）（5）人口统计数据用于公平性评估。基于我们对20多种方法在四种主要范式中的分析，我们揭示了三个重要缺点：尽管最佳方法在mAP@10上达到28.5%，仍然有9%的时间检索到无关结果（困难负样本）。最佳模型在同义词测试中表现出25.1%的性能变化，表明当前CIR技术有显著提升潜力。多图像查询在不同方法中表现较差，差40至70%。为克服我们评估框架发现的新问题，我们提出了一种无需训练的重排序方法，基于现成的MLLM，可以应用于任何现有系统以缩小差距。我们发布了完整的数据集，包括所有图像、查询、注释、检索索引和基准测试代码。
英文摘要：
Composed Image Retrieval (CIR) has made significant progress, yet current benchmarks are limited to single ground-truth answers and lack the annotations needed to evaluate false positive avoidance, robustness and multi-image reasoning. We present PinPoint, a comprehensive real world benchmark with 7,635 queries and 329K relevance judgments across 23 query categories. PinPoint advances the field by providing: (1) multiple correct answers (averaging 9.1 per query) (2) explicit hard negatives, (3) six instruction paraphrases per query for robustness testing, (4) multi-image composition support (13.4% of queries), and (5) demographic metadata for fairness evaluation. Based on our analysis of 20+ methods across 4 different major paradigms, we uncover three significant drawbacks: The best methods while achieving mAP@10 of 28.5%, still retrieves irrelevant results (hard negatives) 9% of the time. The best models also exhibit 25.1% performance variation across paraphrases, indicating significant potential for enhancing current CIR techniques. Multi-image queries performs 40 to 70% worse across different methods. To overcome these new issues uncovered by our evaluation framework, we propose a training-free reranking method based on an off-the-shelf MLLM that can be applied to any existing system to bridge the gap. We release the complete dataset, including all images, queries, annotations, retrieval index, and benchmarking code.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
