Harlequin：面向指代表达理解的色彩驱动合成数据生成
Harlequin: Color-driven Generation of Synthetic Data for Referring Expression Comprehension
AI总结：
本文提出Harlequin框架，通过修改现有注释并据此生成图像，自动合成超过100万条REC查询数据，预训练实验证明该人工数据能有效提升指代表达理解模型性能。
摘要：
指代表达理解（REC）旨在通过自然语言表达识别场景中的特定对象，是视觉语言理解中的一个重要课题。该任务的最先进方法基于深度学习，通常需要昂贵且人工标注的注释。一些工作通过有限监督学习或依赖大型视觉与语言模型来解决该问题。然而，合成标注数据的相关技术开发却被忽视。本文提出了一种新颖的框架，为REC任务生成人工数据，同时考虑文本和视觉两种模态。首先，我们的流程处理现有数据以在注释中创建变体。然后，它使用修改后的注释作为引导生成图像。该流程的结果是一个新数据集，名为Harlequin，包含超过100万条查询。这种方法消除了人工数据收集和标注，实现了可扩展性，并便于生成任意复杂度。我们在Harlequin上预训练了三个REC模型，然后在人工标注数据集上进行微调和评估。实验表明，在人工数据上进行预训练有助于提升性能。
英文摘要：
Referring Expression Comprehension (REC) aims to identify a particular object in a scene by a natural language expression, and is an important topic in visual language understanding. State-of-the-art methods for this task are based on deep learning, which generally requires expensive and manually labeled annotations. Some works tackle the problem with limited-supervision learning or relying on Large Vision and Language Models. However, the development of techniques to synthesize labeled data is overlooked. In this paper, we propose a novel framework that generates artificial data for the REC task, taking into account both textual and visual modalities. At first, our pipeline processes existing data to create variations in the annotations. Then, it generates an image using altered annotations as guidance. The result of this pipeline is a new dataset, called Harlequin, made by more than 1M queries. This approach eliminates manual data collection and annotation, enabling scalability and facilitating arbitrary complexity. We pre-train three REC models on Harlequin, then fine-tuned and evaluated on human-annotated datasets. Our experiments show that the pre-training on artificial data is beneficial for performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
