IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
University of Copenhagen(哥本哈根大学);Mila – Quebec Artificial Intelligence Institute(米拉-魁北克人工智能研究所);University of Cambridge(剑桥大学);TU Darmstadt(达姆施塔特工业大学);New York University(纽约大学);McGill University(麦吉尔大学)
Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, Ivan Vulić
Reliable evaluation benchmarks designed for replicability and comprehensiveness have driven progress in machine learning. Due to the lack of a multilingual benchmark, however, vision-and-language research has mostly focused on English language tasks. To fill this gap, we introduce the Image-Grounded Language Understanding Evaluation benchmark. IGLUE brings together - by both aggregating pre-existing datasets and creating new ones - visual question answering, cross-modal retrieval, grounded reasoning, and grounded entailment tasks across 20 diverse languages. Our benchmark enables the evaluation of multilingual multimodal models for transfer learning, not only in a zero-shot setting, but also in newly defined few-shot learning setups. Based on the evaluation of the available state-of-the-art models, we find that translate-test transfer is superior to zero-shot transfer and that few-shot learning is hard to harness for many tasks. Moreover, downstream performance is partially explained by the amount of available unlabelled textual data for pretraining, and only weakly by the typological distance of target-source languages. We hope to encourage future research efforts in this area by releasing the benchmark to the community.
Anatomical Consistency Distillation and Inconsistency Synthesis for Brain Tumor Segmentation with Missing Modalities
University of Science and Technology of China(中国科学技术大学);Tianjin University(天津大学);Beijing Tiantan Hospital, Capital Medical University(首都医科大学附属北京天坛医院);Hefei Comprehensive National Science Center(合肥综合性国家科学中心)
MultiFly: A Real-World Multimodal Aerial Dataset with Annotation-Efficient Label Transfer and Cross-Modal Semantic Consistency
Fraunhofer Institute IVI(弗劳恩霍夫交通系统研究所);Technical University of Munich(慕尼黑工业大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);University of Cambridge(剑桥大学);Univ. of Applied Sciences Ingolstadt(英戈尔施塔特应用科学大学)
Adversarial Convolutional Networks with Weak Domain-Transfer for Multi-Sequence Cardiac MR Images Segmentation
University of Dundee(邓迪大学);Technical University of Munich(慕尼黑工业大学);Shenzhen Institute of Artificial Intelligence and Robotics for Society(深圳人工智能与机器人研究院);Southern University of Science and Technology(南方科技大学)
SimMAT: Exploring Transferability from Vision Foundation Models to Any Image Modality
Center for Artificial Intelligence and Robotics, HKISI, CAS(中国科学院香港创新研究院人工智能与机器人中心);Princeton University(普林斯顿大学);The Hong Kong Polytechnic University(香港理工大学);The Hong Kong University of Science and Technology(香港科技大学);Nanyang Technological University(南洋理工大学);State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(中国科学院自动化研究所多模态人工智能系统国家重点实验室)
CCSD: Cross-Modal Compositional Self-Distillation for Robust Brain Tumor Segmentation with Missing Modalities
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University(上海智能自主系统研究院,同济大学);College of Biomedical Engineering, Fudan University(生物医学工程学院,复旦大学);School of Medicine, Tongji University(医学院,同济大学);College of Information Engineering, China Jiliang University(信息工程学院,中国嘉兴大学)
Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities
Academy for Engineering and Technology, Fudan University(复旦大学工程技术学院);Jilin Provincial Key Laboratory of Intelligence Science and Engineering(吉林省智能科学与工程重点实验室);Engineering Research Center of AI and Robotics, Ministry of Education(教育部人工智能与机器人工程研究中心)
ACN: Adversarial Co-training Network for Brain Tumor Segmentation with Missing Modalities
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Lenovo Ltd.(联想有限公司);Duke University(杜克大学);University of North Charlotte(北卡罗来纳大学夏洛特分校)
Open-Ended Visual Question Answering by Multi-Modal Domain Adaptation
Northwestern University(西北大学);Futurewei Technologies(华为未来技术研究院);University of Electronic Science and Technology of China(电子科技大学);University of Rochester(罗切斯特大学)
Cross-Modal Few-Shot Learning: a Generative Transfer Learning Framework
National Engineering Research Center for Multimedia Software(国家多媒体软件工程技术研究中心);Institute of Artificial Intelligence(人工智能研究院);School of Computer Science(计算机学院);Wuhan University(武汉大学);Hubei Key Laboratory of Multimedia and Network Communication Engineering(多媒体与网络通信工程湖北省重点实验室);Centre for Frontier AI Research(前沿人工智能研究中心);Agency for Science, Technology and Research(新加坡科技研究局);University of Maryland, College Park(马里兰大学帕克分校);University of Toronto(多伦多大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学)
Multimodal Sentiment Analysis with Missing Modality: A Knowledge-Transfer Approach
College of Computing and Data Science, Nanyang Technological University, Singapore(南洋理工大学计算机与数据科学学院);Singapore University of Social Sciences, Singapore(新加坡社会科学研究大学)
MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
Washington University in St. Louis(华盛顿大学圣路易斯分校);The University of British Columbia(不列颠哥伦比亚大学);Google Research(谷歌研究);Virginia Tech(弗吉尼亚理工大学);University of California, Davis(加州大学戴维斯分校);Sony AI(索尼人工智能);University of California, Berkeley(加州大学伯克利分校)