ObjectCompose：评估基于视觉的模型在对象-背景组合变化下的韧性
ObjectCompose: Evaluating Resilience of Vision-Based Models on Object-to-Background Compositional Changes
AI总结：
该研究提出ObjectCompose方法，在保留对象语义外观的前提下生成多样对象-背景变化，构建多版本标准视觉数据集，评估视觉模型在不同任务下对这类背景变化的鲁棒性。
摘要：
鉴于当前基于视觉的模型采用大规模多模态训练且具备泛化能力，厘清其鲁棒性程度对实际部署至关重要。本研究评估了现有基于视觉的模型针对各类对象-背景上下文变化的韧性。大多数鲁棒性评估方法要么引入合成数据集来改变对象特征（视角、尺度、颜色），要么对真实图像使用图像变换技术（对抗性变化、常见损坏）来模拟分布偏移。近期研究探索了利用大语言模型和扩散模型生成背景变化，但这些方法要么无法对要施加的变化进行控制，要么会扭曲对象语义，不适用于该任务。而我们的方法能够在保留对象原始语义和外观的同时，生成多样的对象-背景变化。为实现这一目标，我们利用文生图、图生文和图生分割模型的生成能力，自动生成广泛的对象-背景变化。我们通过修改文本提示，或是优化文生图模型的潜变量与文本嵌入，来生成自然和对抗性的背景变化。我们制作了多个版本的标准视觉数据集（ImageNet、COCO），要么为图像加入多样且真实的背景，要么在背景中引入颜色、纹理和对抗性变化。我们开展了大量实验，分析基于视觉的模型在不同任务中应对对象-背景上下文变化的鲁棒性。代码见https://github.com/Muhammad-Huzaifaa/ObjectCompose。
英文摘要：
Given the large-scale multi-modal training of recent vision-based models and their generalization capabilities, understanding the extent of their robustness is critical for their real-world deployment. In this work, we evaluate the resilience of current vision-based models against diverse object-to-background context variations. The majority of robustness evaluation methods have introduced synthetic datasets to induce changes to object characteristics (viewpoints, scale, color) or utilized image transformation techniques (adversarial changes, common corruptions) on real images to simulate shifts in distributions. Recent works have explored leveraging large language models and diffusion models to generate changes in the background. However, these methods either lack in offering control over the changes to be made or distort the object semantics, making them unsuitable for the task. Our method, on the other hand, can induce diverse object-to-background changes while preserving the original semantics and appearance of the object. To achieve this goal, we harness the generative capabilities of text-to-image, image-to-text, and image-to-segment models to automatically generate a broad spectrum of object-to-background changes. We induce both natural and adversarial background changes by either modifying the textual prompts or optimizing the latents and textual embedding of text-to-image models. We produce various versions of standard vision datasets (ImageNet, COCO), incorporating either diverse and realistic backgrounds into the images or introducing color, texture, and adversarial changes in the background. We conduct extensive experiments to analyze the robustness of vision-based models against object-to-background context variations across diverse tasks. Code https://github.com/Muhammad-Huzaifaa/ObjectCompose.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
