Recent cross-lingual cross-modal works attempt to extend Vision-Language Pre-training (VLP) models to non-English inputs and achieve impressive performance. However, these models focus only on understanding tasks utilizing encoder-only architecture. In this paper, we propose ERNIE-UniX2, a unified cross-lingual cross-modal pre-training framework for both generation and understanding tasks. ERNIE-UniX2 integrates multiple pre-training paradigms (e.g., contrastive learning and language modeling) based on encoder-decoder architecture and attempts to learn a better joint representation across languages and modalities. Furthermore, ERNIE-UniX2 can be seamlessly fine-tuned for varieties of generation and understanding downstream tasks. Pre-trained on both multilingual text-only and image-text datasets, ERNIE-UniX2 achieves SOTA results on various cross-lingual cross-modal generation and understanding tasks such as multimodal machine translation and multilingual visual question answering.
Embedded Heterogeneous Attention Transformer for Cross-lingual Image Captioning
School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
A Benchmark for Multi-Lingual Vision-Language Learning in Remote Sensing Image Captioning
School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University（西北工业大学人工智能、光学与电子学院（iOPEN））；Northwest Institute of Nuclear Technology（西北核技术研究所）
Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameters
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机科学学院，北京大学）；WeChat AI, Tencent Inc., China（腾讯公司，中国）；The University of Hong Kong（香港大学）
Modelling Visual Semantics via Image Captioning to extract Enhanced Multi-Level Cross-Modal Semantic Incongruity Representation with Attention for Multimodal Sarcasm Detection
Dual-view Curricular Optimal Transport for Cross-lingual Cross-modal Retrieval
Zhejiang Gongshang University（浙江工商大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Alibaba Group（阿里巴巴集团）；Binjiang Institute of Zhejiang University（浙江大学滨江研究院）；Hefei University of Technology（合肥工业大学）
M$^3$-Med: A Benchmark for Multi-lingual, Multi-modal, and Multi-hop Reasoning in Medical Instructional Video Understanding
Beijing Institute of Technology（北京理工大学）；The Hong Kong Polytechnic University（香港理工大学）；Chinese Academy of Sciences（中国科学院）；Air Force Hospital of Southern Theater Command of PLA（中国人民解放军南部战区空军医院）；Shenzhen Traditional Chinese Medicine Hospital（深圳中医药医院）
MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generation
Stability AI（Stability AI公司）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；TU Darmstadt（达姆施塔特工业大学）；Aleph Alpha（Aleph Alpha公司）；University of Texas（得克萨斯大学）；LAION（大规模人工智能开放网络（LAION））
Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Zhongke Fanyu Technology Co., Ltd（中科泛语科技有限公司）
Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias
The Ohio State University（俄亥俄州立大学）；Imperial College London（伦敦帝国学院）；The Hong Kong University of Science and Technology（香港科技大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Peking University（北京大学）
Semnan University（塞姆南大学）；Institute for Research in Fundamental Sciences (IPM)（基础科学研究所（IPM））；Universitat de Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）
零样本学习（Zero-Shot Learning，简称ZRL）近年来发展迅速。为了克服手语识别（SLR）中的注释瓶颈，我们探索了零镜头手语识别（Zero-Shot SLR）的想法，没有注释的视觉示例，通过利用它们的文本描述。通过这种方式，我们提出了一种多模态零镜头手语识别（Zero-Shot Sign Language Recognition，ZSLR）模型，该模型利用了与基于手势的特征融合的深度特征的互补功能。基于transformer的模型和C3 D模型分别用于手部检测和深度特征提取。为了在基于特征的维度和深度特征之间进行权衡，我们在长短期记忆（LSTM）网络之上使用了自动编码器（AE）。最后，一个语义空间被用来映射的视觉特征的类标签的语言嵌入，通过双向编码器表示从Transformers（BERT）模型实现。四个大规模数据集，RKS-PERSIANSIGN，第一人称，ASLVID和isoGD的结果，显示了该模型的优越性相比，国家的最先进的替代品在SLR。
Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models
Mohamed Bin Zayed University of AI, UAE（穆罕默德·本·扎耶德人工智能大学，阿联酋）；Khalifa University, UAE（卡比拉大学，阿联酋）；Australian National University, Australia（澳大利亚国立大学，澳大利亚）
Targeted Interpretable Safety Neuron Enhancement for Multilingual Vision-Language Large Models
Nanjing University of Science and Technology（南京理工大学）；National University of Singapore（新加坡国立大学）；Beihang University（北京航空航天大学）；Nanjing Forestry University（南京林业大学）
LDP: Generalizing to Multilingual Visual Information Extraction by Language Decoupled Pretraining
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Nankai University（南开大学）；University of Chinese Academy of Sciences（中国科学院大学）
