问：如何使大型视觉-语言模型专门化于数据稀缺的 VQA 任务？答：在无标签图像上进行自训练！
Q: How to Specialize Large Vision-Language Models to Data-Scarce VQA Tasks? A: Self-Train on Unlabeled Images!
AI总结：
本文提出 SelTDA，利用目标数据集构建教师模型为无标签图像生成问答伪标签，并据此微调 VLM，在数据稀缺的专门 VQA 任务上提升鲁棒性、泛化性与数值推理能力。
摘要：
在大规模预训练之后，于目标数据集上微调大型视觉-语言模型（VLM）是视觉问答（VQA）中的主流范式。用于专门任务（如基于知识的 VQA）或非自然图像领域中 VQA 的数据集，比用于通用 VQA 的数据集小几个数量级。尽管为专门任务或领域收集额外标签可能具有挑战性，但无标签图像通常是可获得的。我们提出 SelTDA（Self-Taught Data Augmentation，自学数据增强），这是一种在小规模 VQA 数据集上微调大型 VLM 的策略。SelTDA 使用 VLM 和目标数据集构建一个教师模型，该模型能够直接仅以图像为条件生成问答伪标签，从而使我们能够对无标签图像进行伪标注。随后，SelTDA 在原始数据集与新近伪标注图像共同增强得到的数据上微调初始 VLM。我们描述了一系列实验，结果表明，我们的自学数据增强提高了模型对对抗性搜索问题、反事实示例和改写的鲁棒性，改善了领域泛化，并使数值推理技能得到更好的保持。所提出的策略不需要额外标注或架构修改，并且兼容任何现代编码器-解码器多模态 Transformer。代码可在 https://github.com/codezakh/SelTDA 获取。
英文摘要：
Finetuning a large vision language model (VLM) on a target dataset after large scale pretraining is a dominant paradigm in visual question answering (VQA). Datasets for specialized tasks such as knowledge-based VQA or VQA in non natural-image domains are orders of magnitude smaller than those for general-purpose VQA. While collecting additional labels for specialized tasks or domains can be challenging, unlabeled images are often available. We introduce SelTDA (Self-Taught Data Augmentation), a strategy for finetuning large VLMs on small-scale VQA datasets. SelTDA uses the VLM and target dataset to build a teacher model that can generate question-answer pseudolabels directly conditioned on an image alone, allowing us to pseudolabel unlabeled images. SelTDA then finetunes the initial VLM on the original dataset augmented with freshly pseudolabeled images. We describe a series of experiments showing that our self-taught data augmentation increases robustness to adversarially searched questions, counterfactual examples and rephrasings, improves domain generalization, and results in greater retention of numerical reasoning skills. The proposed strategy requires no additional annotations or architectural modifications, and is compatible with any modern encoder-decoder multimodal transformer. Code available at https://github.com/codezakh/SelTDA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
