Babel-ImageNet：视觉与语言表示的大规模多语言评估
Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representations
AI总结：
本文提出覆盖100种语言的 Babel-ImageNet 多语言零样本图像分类基准，并评估11个多语言 CLIP 模型，揭示语言资源差异及分类与检索性能的相关性。
摘要：
针对每种模态采用独立编码器的视觉与语言（VL）模型（例如 CLIP），已成为零样本图像分类和图文检索的首选模型。然而，由于多语言基准在可用性上有限，这些模型大多只在英语中接受评估。我们提出 Babel-ImageNet，这是一个大规模多语言基准，提供 ImageNet 标签到 100 种语言的（部分）翻译，并且其构建不依赖机器翻译或人工标注。相反，我们通过共享的 WordNet 同义词集，将这些标签链接到 BabelNet 这一大规模多语言词汇语义网络，从而自动获得可靠翻译。我们在该基准上评估了 11 个公开的多语言 CLIP 模型的零样本图像分类（ZS-IC）性能，结果表明，英语 ImageNet 性能与德语或中文等高资源语言性能之间存在显著差距，而僧伽罗语或老挝语等低资源语言的差距更大。关键在于，我们证明模型的 ZS-IC 性能与其在图文检索中的性能高度相关，从而验证了可使用 Babel-ImageNet 为绝大多数没有黄金图文数据的语言评估多语言模型。最后，我们表明，通过参数高效的特定语言训练，可以大幅提升多语言 CLIP 在低资源语言上的性能。我们公开了代码和数据：https://github.com/gregor-ge/Babel-ImageNet。
英文摘要：
Vision-and-language (VL) models with separate encoders for each modality (e.g., CLIP) have become the go-to models for zero-shot image classification and image-text retrieval. They are, however, mostly evaluated in English as multilingual benchmarks are limited in availability. We introduce Babel-ImageNet, a massively multilingual benchmark that offers (partial) translations of ImageNet labels to 100 languages, built without machine translation or manual annotation. We instead automatically obtain reliable translations by linking them -- via shared WordNet synsets -- to BabelNet, a massively multilingual lexico-semantic network. We evaluate 11 public multilingual CLIP models on zero-shot image classification (ZS-IC) on our benchmark, demonstrating a significant gap between English ImageNet performance and that of high-resource languages (e.g., German or Chinese), and an even bigger gap for low-resource languages (e.g., Sinhala or Lao). Crucially, we show that the models' ZS-IC performance highly correlates with their performance in image-text retrieval, validating the use of Babel-ImageNet to evaluate multilingual models for the vast majority of languages without gold image-text data. Finally, we show that the performance of multilingual CLIP can be drastically improved for low-resource languages with parameter-efficient language-specific training. We make our code and data publicly available: \url{https://github.com/gregor-ge/Babel-ImageNet}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
