OmniDiff: 一个全面的细粒度图像差异描述基准
OmniDiff: A Comprehensive Benchmark for Fine-grained Image Difference Captioning
AI总结：
OmniDiff提出一个全面的细粒度图像差异描述基准，结合多模态大语言模型与插拔式多尺度差异感知模块，提升跨场景差异识别性能。
摘要：
图像差异描述（IDC）旨在生成图像对之间细微差异的自然语言描述，需要精确的视觉变化定位和连贯的语义表达。尽管近年来取得了进展，现有数据集往往缺乏广度和深度，限制了其在复杂和动态环境中的应用：（1）从广度来看，当前数据集受限于特定场景中有限的物体变化；（2）从深度来看，先前的基准往往提供过于简化的描述。为了解决这些挑战，我们引入了OmniDiff，一个包含324种多样场景的数据集，涵盖现实复杂环境和3D合成设置，具有平均长度为60词的细粒度人工注释，覆盖12种不同的变化类型。在此基础上，我们提出了M$^3$Diff，一种通过插拔式多尺度差异感知（MDP）模块增强的多模态大语言模型。该模块提高了模型准确识别和描述图像间差异的能力，同时保持基础模型的泛化能力。在添加OmniDiff数据集后，M$^3$Diff在多个基准上实现了最先进的性能，包括Spot-the-Diff、IEdit、CLEVR-Change、CLEVR-DC和OmniDiff，展示了显著优于现有方法的跨场景差异识别准确性。该数据集、代码和模型将向公众开放，以支持进一步研究。
英文摘要：
Image Difference Captioning (IDC) aims to generate natural language descriptions of subtle differences between image pairs, requiring both precise visual change localization and coherent semantic expression. Despite recent advancements, existing datasets often lack breadth and depth, limiting their applicability in complex and dynamic environments: (1) from a breadth perspective, current datasets are constrained to limited variations of objects in specific scenes, and (2) from a depth perspective, prior benchmarks often provide overly simplistic descriptions. To address these challenges, we introduce OmniDiff, a comprehensive dataset comprising 324 diverse scenarios-spanning real-world complex environments and 3D synthetic settings-with fine-grained human annotations averaging 60 words in length and covering 12 distinct change types. Building on this foundation, we propose M$^3$Diff, a MultiModal large language model enhanced by a plug-and-play Multi-scale Differential Perception (MDP) module. This module improves the model's ability to accurately identify and describe inter-image differences while maintaining the foundational model's generalization capabilities. With the addition of the OmniDiff dataset, M$^3$Diff achieves state-of-the-art performance across multiple benchmarks, including Spot-the-Diff, IEdit, CLEVR-Change, CLEVR-DC, and OmniDiff, demonstrating significant improvements in cross-scenario difference recognition accuracy compared to existing methods. The dataset, code, and models will be made publicly available to support further research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
