视觉和语言预训练在学习视觉和语言之间的多模态表征方面取得了令人印象深刻的成功。为了将这一成功推广到非英语语言，我们引入了UC 2，这是第一个用于跨语言跨模态表示学习的机器翻译增强框架。为了解决图像数据集的多语言字幕稀缺问题，我们首先通过机器翻译（MT）用其他语言增强现有的仅英语的数据集。然后，我们将标准的掩蔽语言建模和图像-文本匹配训练目标扩展到多语言设置，其中通过共享视觉上下文（即，使用图像作为枢轴）捕获不同语言之间的对齐。为了促进图像和所有感兴趣语言的联合嵌入空间的学习，我们进一步提出了两个新的预训练任务，即Masked Region-to-Token Modeling（MRTM）和Visual Translation Language Modeling（VTLM），利用MT增强的翻译数据。对多语言图像-文本检索和多语言视觉问答基准测试的评估表明，我们提出的框架在各种非英语基准测试上达到了新的最先进水平，同时保持了与英语任务单语预训练模型相当的性能。
英文摘要：
Vision-and-language pre-training has achieved impressive success in learning multimodal representations between vision and language. To generalize this success to non-English languages, we introduce UC2, the first machine translation-augmented framework for cross-lingual cross-modal representation learning. To tackle the scarcity problem of multilingual captions for image datasets, we first augment existing English-only datasets with other languages via machine translation (MT). Then we extend the standard Masked Language Modeling and Image-Text Matching training objectives to multilingual setting, where alignment between different languages is captured through shared visual context (i.e, using image as pivot). To facilitate the learning of a joint embedding space of images and all languages of interest, we further propose two novel pre-training tasks, namely Masked Region-to-Token Modeling (MRTM) and Visual Translation Language Modeling (VTLM), leveraging MT-enhanced translated data. Evaluation on multilingual image-text retrieval and multilingual visual question answering benchmarks demonstrates that our proposed framework achieves new state-of-the-art on diverse non-English benchmarks while maintaining comparable performance to monolingual pre-trained models on English tasks.
Embedded Heterogeneous Attention Transformer for Cross-lingual Image Captioning
School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
A Benchmark for Multi-Lingual Vision-Language Learning in Remote Sensing Image Captioning
School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University（西北工业大学人工智能、光学与电子学院（iOPEN））；Northwest Institute of Nuclear Technology（西北核技术研究所）
Dual-view Curricular Optimal Transport for Cross-lingual Cross-modal Retrieval
Zhejiang Gongshang University（浙江工商大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Alibaba Group（阿里巴巴集团）；Binjiang Institute of Zhejiang University（浙江大学滨江研究院）；Hefei University of Technology（合肥工业大学）
M$^3$-Med: A Benchmark for Multi-lingual, Multi-modal, and Multi-hop Reasoning in Medical Instructional Video Understanding
Beijing Institute of Technology（北京理工大学）；The Hong Kong Polytechnic University（香港理工大学）；Chinese Academy of Sciences（中国科学院）；Air Force Hospital of Southern Theater Command of PLA（中国人民解放军南部战区空军医院）；Shenzhen Traditional Chinese Medicine Hospital（深圳中医药医院）
MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generation
Stability AI（Stability AI公司）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；TU Darmstadt（达姆施塔特工业大学）；Aleph Alpha（Aleph Alpha公司）；University of Texas（得克萨斯大学）；LAION（大规模人工智能开放网络（LAION））
Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias
The Ohio State University（俄亥俄州立大学）；Imperial College London（伦敦帝国学院）；The Hong Kong University of Science and Technology（香港科技大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Peking University（北京大学）
Universal Weighting Metric Learning for Cross-Modal Matching
Center for Future Media（未来媒体中心）；School of Computer Science and Engineering（计算机科学与工程学院）；University of Electronic Science and Technology of China（电子科技大学）
Dual-Space Modality Consistency Learning for Universal Cross-Modal Re-Identification
School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；School of Humanities and Social Sciences, Beihang University（北京航空航天大学人文与社会科学高等研究院）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Informatics, Xiamen University（厦门大学信息学院）
Dealing with All-stage Missing Modality: Towards A Universal Model with Robust Reconstruction and Personalization
National University of Singapore（新加坡国立大学）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）；National Neuroscience Institute（国家神经科学研究所）；Duke-NUS Medical School（杜克-新加坡国立大学医学院）
MedUniSeg: 2D and 3D Medical Image Segmentation via a Prompt-driven Universal Model
Northwestern Polytechnical University（西北工业大学）；Zhejiang University（浙江大学）；The University of Adelaide（阿德莱德大学）；Australian Institute for Machine Learning (AIML)（澳大利亚机器学习研究所）；Research & Development Institute of Northwestern Polytechnical University in Shenzhen（西北工业大学深圳研究院）；Ningbo Institute of Northwestern Polytechnical University（西北工业大学宁波研究院）
