PaLI-X: On Scaling up a Multilingual Vision and Language Model
Google Research（谷歌研究院）
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, Siamak Shake… 展开作者
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, Siamak Shakeri, Mostafa Dehghani, Daniel Salz, Mario Lucic, Michael Tschannen, Arsha Nagrani, Hexiang Hu, Mandar Joshi, Bo Pang, Ceslee Montgomery, Paulina Pietrzyk, Marvin Ritter, AJ Piergiovanni, Matthias Minderer, Filip Pavetic, Austin Waters, Gang Li, Ibrahim Alabdulmohsin, Lucas Beyer, Julien Amelot, Kenton Lee, Andreas Peter Steiner, Yang Li, Daniel Keysers, Anurag Arnab, Yuanzhong Xu, Keran Rong, Alexander Kolesnikov, Mojtaba Seyedhosseini, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, Radu Soricut
We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture. Our model achieves new levels of performance on a wide-range of varied and complex tasks, including multiple image-based captioning and question-answering tasks, image-based document understanding and few-shot (in-context) learning, as well as object detection, video question answering, and video captioning. PaLI-X advances the state-of-the-art on most vision-and-language benchmarks considered (25+ of them). Finally, we observe emerging capabilities, such as complex counting and multilingual object detection, tasks that are not explicitly in the training mix.
MITRA: A Large-Scale Parallel Corpus and Multilingual Pretrained Language Model for Machine Translation and Semantic Retrieval for Pāli, Sanskrit, Buddhist Chinese, and Tibetan
Center for Integrated Japanese Studies, Tohoku University（东京东京大学综合日语研究机构）；University of California, Berkeley（加州大学伯克利分校）；Berkeley Artificial Intelligence Research (BAIR)（伯克利人工智能研究（BAIR））
VFA: Empowering Multilingual MLLMs via Vision-Free Adaptation
Southern University of Science and Technology（南方科技大学）；Microsoft Research Asia（微软亚洲研究院）；Shanghai University of Finance and Economics（上海财经大学）；Peking University（北京大学）
TowerVision: Understanding and Improving Multilinguality in Vision-Language Models
Instituto Superior Técnico, Universidade de Lisboa（里斯本大学技术高级学院）；Instituto de Telecomunicações（电信研究所）；Carnegie Mellon University（卡内基梅隆大学）；Sword Health（Sword健康）；TransPerfect；MICS, CentraleSupélec, Université Paris-Saclay（MICS，中央圣艾尔布里大学，巴黎萨克雷大学）；ELLIS Unit Lisbon（里斯本ELLIS单位）
Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models
Mohamed Bin Zayed University of AI, UAE（穆罕默德·本·扎耶德人工智能大学，阿联酋）；Khalifa University, UAE（卡比拉大学，阿联酋）；Australian National University, Australia（澳大利亚国立大学，澳大利亚）
MUDIDI: A Two-Stage Framework for Multilingual Dictionary Digitization with Language Models
School of Computing and Information Systems, The University of Melbourne（墨尔本大学计算与信息系统学院）；Melbourne School of Psychological Sciences, The University of Melbourne（墨尔本大学墨尔本心理科学学院）；LILT
Research Centre for Cognitive Science & Artificial Intelligence（认知科学与人工智能研究中心）；Department of Computational Cognitive Science（计算认知科学系）；Tilburg University（蒂尔堡大学）
Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptation
Alibaba International Digital Commerce Group（阿里巴巴国际数字商务集团）；Universität Hamburg, Department of Informatics（汉堡大学信息学院）；Nanyang Technological University（南洋理工大学）；Monash University（莫纳什大学）
A Pilot Study on Curator-Guided Multilingual Art Description for Blind and Low-Vision Audiences with Small Vision-Language Models
Technical University of Munich（慕尼黑技术大学）；Foundation for Research and Technology -- Hellas（希腊研究与技术基金会）；National University of Science and Technology Politehnica Bucharest（布加勒斯特政治技术科学与技术国家大学）
A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scene
School of Geography and Information Engineering, China University of Geosciences（中国地质大学地理与信息工程学院）；Meituan（美团）；National Engineering Research Center for Geographic Information System, China University of Geosciences（中国地质大学国家地理信息系统工程技术研究中心）
Vision-Language Models are Fragile Multilingual Associators
Manipal University Jaipur（斋浦尔马尼帕尔大学）；University of North Carolina Charlotte（北卡罗来纳大学夏洛特分校）；University of Salford（索尔福德大学）；University of Manchester（曼彻斯特大学）；Indian Statistical Institute Kolkata（印度统计研究所加尔各答分所）
Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation
Cohere For AI（科here人工智能研究院）；University of Copenhagen（哥本哈根大学）；Institute of Computer Sciences, CONICET & Universidad de Buenos Aires（阿根廷科学研究理事会与布宜诺斯艾利斯大学计算机科学研究所）；Cohere For AI Community（科here人工智能社区）；University of California, Santa Barbara（加州大学圣巴巴拉分校）；Emory University（埃默里大学）；M2ai.in；Taras Shevchenko National University of Kyiv（基辅塔拉斯舍甫琴科国立大学）；Aalborg University（奥尔堡大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；ScaDS.AI, TU Dresden（德累斯顿工业大学ScaDS.AI）；T-Tech；Lviv Polytechnic National University（利沃夫国立理工大学）；HSE University (Higher School of Economics)（高等经济大学）；RAFT；ETH Zürich（苏黎世联邦理工学院）；University of São Paulo（圣保罗大学）；National University Philippines（菲律宾国立大学）；Federal University of São Carlos（圣卡洛斯联邦大学）；SomosNLP；Cisco（思科公司）；Comenius University in Bratislava（布拉迪斯拉发夸美纽斯大学）；Mila, University of Montreal（蒙特利尔大学Mila研究所）；Uppsala University（乌普萨拉大学）；University of Bath（巴斯大学）；Pioneer Center for AI（人工智能先锋中心）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）
LDP: Generalizing to Multilingual Visual Information Extraction by Language Decoupled Pretraining
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Nankai University（南开大学）；University of Chinese Academy of Sciences（中国科学院大学）
Mitigating Multilingual Hallucination in Large Vision-Language Models
School of Computer Science & Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）；School of Computing Science and Technology, Fudan University（复旦大学计算机科学技术学院）；Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；School of Computing Science and Technology, Zhejiang Gongshang University（浙江工商大学计算机科学与技术学院）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程学系）
Targeted Interpretable Safety Neuron Enhancement for Multilingual Vision-Language Large Models
Nanjing University of Science and Technology（南京理工大学）；National University of Singapore（新加坡国立大学）；Beihang University（北京航空航天大学）；Nanjing Forestry University（南京林业大学）
Benchmarking and Boosting Multilingual Capabilities of LVLMs via OCR-Centric Reinforcement Learning
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；Peking University（北京大学）；Sun Yat-Sen University（中山大学）；Chinese University of Hong Kong（香港中文大学）
Multilingual Vision-Language Pre-training for the Remote Sensing Domain
INESC-ID；Instituto Superior Técnico（高等技术学院）；University of Lisbon（里斯本大学）；NOVA-LINCS；Faculty of Science and Technology（科学与技术学院）；NOVA University（NOVA大学）；EPFL（洛桑联邦理工学院）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；LUMLIS (Lisbon ELLIS Unit)（LUMLIS（里斯本ELLIS单元））
MIRACL-VISION: A Large, multilingual, visual document retrieval benchmark
The Thørväld Group（Thørväld集团）；Inria Paris-Rocquencourt（巴黎-罗克琴库尔Inria）；Rajiv Gandhi University Doimukh（拉朱·甘地大学多伊穆克）；Tsinghua University（清华大学）；Palmer Research Laboratories（Palmer研究实验室）；The Kumquat Consortium（Kumquat联盟）
