通过翻译泛化实现多语言图像描述条件涌现的缩放定律
Scaling Laws for Conditional Emergence of Multilingual Image Captioning via Generalization from Translation
AI总结：
针对视觉语言模型跨语言跨任务迁移的数据稀缺问题，研究编码器-解码器Transformer VLMs的多语言泛化能力，发现未见过的任务-语言对的间接学习遵循缩放定律，且该定律可扩展到下游任务并取得竞争性性能。
摘要：
跨语言、跨任务迁移面临任务特定数据稀缺的挑战，随着语言支持的增加，这一挑战愈发严峻，在视觉语言模型（VLMs）中更是如此。我们研究编码器-解码器Transformer VLMs的多语言泛化能力，以实现仅在翻译任务中遇到的语言的零样本图像描述。在此设置中，编码器必须学习生成可泛化的、任务感知的潜在视觉表示，通过插入的交叉注意力层指导解码器。为分析缩放行为，我们使用不同计算预算在合成数据集上训练基于Florence-2和Gemma-2的模型（0.4B至11.2B参数）。尽管数据集中所有语言都有图像对齐的翻译，但只有其中一个子集包含图像描述。值得注意的是，我们表明即使该语言仅出现在翻译任务中，使用语言前缀也能涌现出描述能力。我们发现未见过的任务-语言对的间接学习遵循由模型多语言性、模型大小和见过的训练样本主导的缩放定律。最后，我们证明这些缩放定律可扩展到下游任务，通过微调在多模态机器翻译（Multi30K、CoMMuTE）、词汇消歧（CoMMuTE）和图像描述（Multi30K、XM3600、COCO Karpathy）中取得竞争性性能。
英文摘要：
Cross-lingual, cross-task transfer is challenged by task-specific data scarcity, which becomes more severe as language support grows and is further amplified in vision-language models (VLMs). We investigate multilingual generalization in encoder-decoder transformer VLMs to enable zero-shot image captioning in languages encountered only in the translation task. In this setting, the encoder must learn to generate generalizable, task-aware latent vision representations to instruct the decoder via inserted cross-attention layers. To analyze scaling behavior, we train Florence-2 based and Gemma-2 based models (0.4B to 11.2B parameters) on a synthetic dataset using varying compute budgets. While all languages in the dataset have image-aligned translations, only a subset of them include image captions. Notably, we show that captioning can emerge using a language prefix, even when this language only appears in the translation task. We find that indirect learning of unseen task-language pairs adheres to scaling laws that are governed by the multilinguality of the model, model size, and seen training samples. Finally, we demonstrate that the scaling laws extend to downstream tasks, achieving competitive performance through fine-tuning in multimodal machine translation (Multi30K, CoMMuTE), lexical disambiguation (CoMMuTE), and image captioning (Multi30K, XM3600, COCO Karpathy).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
