AI 中文总结
研究人类视觉相似性判断的上下文依赖问题，通过引入大规模数据集微调VLM，产生能捕捉多种视觉相似性意义的TPIPS度量，该度量与人类感知更契合，还在多种任务中开启新功能。
AI 中文摘要
人类视觉相似性判断依赖于上下文。例如，两张图像可能形状相似但颜色不同。然而，现有的感知相似性度量将这些细微差别归结为单个标量值，没有针对特定方面进行条件设定的机制。为弥合这一差距，我们引入了一个关于图像三元组的人类相似性判断的大规模数据集，每个三元组在多个自由形式的相似性语义方面进行注释。对一系列前沿视觉语言模型（VLM）进行基准测试发现，与人类注释者的共识相比存在相当大的性能差距。利用我们的数据，我们微调了一个VLM以产生我们的文本提示图像感知相似性（TPIPS）度量，根据指定的文本提示捕捉多种视觉相似性意义。我们证明TPIPS与人类感知更紧密对齐，并且在训练分布之外可靠地泛化。最后，我们表明TPIPS在文本引导检索、组合搜索和生成模型的细粒度评估中开启了新功能。我们的代码、数据和训练模型可在这个https网址获取。
英文摘要
Human visual similarity judgments are context-dependent. For example, two images may be similar in shape but distinct in color. Existing perceptual similarity metrics, however, collapse these nuances into a single scalar value, offering no mechanism to condition on specific aspects. To bridge this gap, we introduce a large-scale dataset of human similarity judgments over image triplets, where each triplet is annotated across multiple, free-form semantic aspects of similarity. Benchmarking a broad range of frontier vision-language models (VLMs) reveals a considerable performance gap compared to human annotators' consensus. Leveraging our data, we fine-tune a VLM to produce our Text-Prompted Image Perceptual Similarity (TPIPS) metric, capturing multiple senses of visual similarity depending on the specified text prompt. We demonstrate that TPIPS aligns more closely with human perception and generalizes reliably beyond the training distribution. Finally, we show that TPIPS unlocks new capabilities in text-guided retrieval, compositional search, and the fine-grained evaluation of generative models. Our code, data, and trained models are at https://peterwang512.github.io/TPIPS
CommentsProject Webpage: https://peterwang512.github.io/TPIPS Code: https://github.com/adobe-research/TPIPS
