CtrlSynth：用于数据高效多模态学习的可控图文合成
CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learning
AI总结：
本文提出可控图文合成框架CtrlSynth，通过分解图像语义为基本元素并应用用户控制策略重组，生成多样化合成数据，显著提升CLIP模型在零样本分类等任务上的性能。
摘要：
预训练鲁棒的视觉或多模态基础模型（例如CLIP）依赖于大规模数据集，这些数据集可能存在噪声、潜在未对齐以及长尾分布问题。以往的工作通过生成合成样本来扩充数据集，取得了可喜的成果。然而，它们仅支持特定领域的临时用例（例如仅图像或仅文本，而非两者），并且由于缺乏对合成过程的细粒度控制，在数据多样性方面受到限制。在本文中，我们设计了一个可控的图文合成流程CtrlSynth，用于数据高效且鲁棒的多模态学习。核心思想是将图像的视觉语义分解为基本元素，应用用户指定的控制策略（例如移除、添加或替换操作），并将它们重新组合以合成图像或文本。CtrlSynth中的分解与重组特性允许用户通过定义自定义控制策略来操作基本元素，从而以细粒度方式控制数据合成。CtrlSynth利用预训练基础模型（如大型语言模型或扩散模型）的能力来推理和重组基本元素，使得合成样本自然且组合方式多样。CtrlSynth是一个闭环、免训练且模块化的框架，易于支持不同的预训练模型。通过在跨越不同视觉和视觉语言任务的31个数据集上进行广泛实验，我们表明CtrlSynth显著提升了CLIP模型的零样本分类、图文检索和组合推理性能。
英文摘要：
Pretraining robust vision or multimodal foundation models (e.g., CLIP) relies on large-scale datasets that may be noisy, potentially misaligned, and have long-tail distributions. Previous works have shown promising results in augmenting datasets by generating synthetic samples. However, they only support domain-specific ad hoc use cases (e.g., either image or text only, but not both), and are limited in data diversity due to a lack of fine-grained control over the synthesis process. In this paper, we design a \emph{controllable} image-text synthesis pipeline, CtrlSynth, for data-efficient and robust multimodal learning. The key idea is to decompose the visual semantics of an image into basic elements, apply user-specified control policies (e.g., remove, add, or replace operations), and recompose them to synthesize images or texts. The decompose and recompose feature in CtrlSynth allows users to control data synthesis in a fine-grained manner by defining customized control policies to manipulate the basic elements. CtrlSynth leverages the capabilities of pretrained foundation models such as large language models or diffusion models to reason and recompose basic elements such that synthetic samples are natural and composed in diverse ways. CtrlSynth is a closed-loop, training-free, and modular framework, making it easy to support different pretrained models. With extensive experiments on 31 datasets spanning different vision and vision-language tasks, we show that CtrlSynth substantially improves zero-shot classification, image-text retrieval, and compositional reasoning performance of CLIP models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
