arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2201.11732cs.CLcs.CV
收藏

IGLUE:跨模式、任务和语言的迁移学习基准

IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages

University of Copenhagen(哥本哈根大学);Mila – Quebec Artificial Intelligence Institute(米拉-魁北克人工智能研究所);University of Cambridge(剑桥大学);TU Darmstadt(达姆施塔特工业大学);New York University(纽约大学);McGill University(麦吉尔大学)

Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, Ivan Vulić

摘要:

为可复制性和全面性设计的可靠评估基准推动了机器学习的进步。然而,由于缺乏多语言基准,视觉和语言研究主要集中在英语语言任务上。为了填补这一空白,我们引入了基于图像的语言理解评估基准。IGLUE通过聚合已有的数据集和创建新的数据集,汇集了20种不同语言的可视化问答、跨模态检索、接地推理和接地蕴涵任务。我们的基准能够评估迁移学习的多语言多模态模型,不仅在zero-shot设置中,而且在新定义的少样本学习设置中。基于现有的最先进的模型的评估,我们发现,测试迁移优于zero-shot迁移和少样本学习是很难利用许多任务。此外,下游的性能部分解释的数量可用的未标记的文本数据进行预训练,只有微弱的目标-源语言的类型距离。我们希望通过向社区发布基准来鼓励未来在这一领域的研究工作。

英文摘要:

Reliable evaluation benchmarks designed for replicability and comprehensiveness have driven progress in machine learning. Due to the lack of a multilingual benchmark, however, vision-and-language research has mostly focused on English language tasks. To fill this gap, we introduce the Image-Grounded Language Understanding Evaluation benchmark. IGLUE brings together - by both aggregating pre-existing datasets and creating new ones - visual question answering, cross-modal retrieval, grounded reasoning, and grounded entailment tasks across 20 diverse languages. Our benchmark enables the evaluation of multilingual multimodal models for transfer learning, not only in a zero-shot setting, but also in newly defined few-shot learning setups. Based on the evaluation of the available state-of-the-art models, we find that translate-test transfer is superior to zero-shot transfer and that few-shot learning is hard to harness for many tasks. Moreover, downstream performance is partially explained by the amount of available unlabelled textual data for pretraining, and only weakly by the typological distance of target-source languages. We hope to encourage future research efforts in this area by releasing the benchmark to the community.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑