Contrastive Language-Image Pretraining (CLIP) has been widely used for crossmodal information retrieval and multimodal understanding tasks. However, CLIP models are mainly optimized for crossmodal vision-language tasks and underperform in single-mode text tasks. Moreover, these models are often trained on English datasets and therefore lack multilingual understanding. Additionally, from a visual understanding perspective, previous CLIP-based models exhibit insufficient understanding of visually rich documents. In this work, we propose jina-clip-v2, a contrastive vision-language model trained on text pairs, triplets and image-text pairs via a multi-task and multi-stage contrastive learning paradigm in order to support both text-only and crossmodal tasks. We employ a multilingual text encoder and expand the training dataset to include multilingual texts from 29 non-English languages, including Hindi, Chinese, German, French, and others, as well as images of visually rich documents. We evaluate the model's performance and show that jina-clip-v2 achieves notable improvements over state-of-the-art CLIP-based models in zero-shot text-only retrieval, semantic textual similarity, and crossmodal retrieval tasks in both English and multilingual settings. jina-clip-v2 also provides for flexibility in embedding dimensionality, enabling users to select the granularity of the representations. jina-clip-v2 is publicly available at https://huggingface.co/jinaai/jina-clip-v2.
Language-Conditioned Visual Grounding with CLIP Multilingual
1 Department of Computer Applications in Science \& Engineering, BARCELONA Supercomputing Center, Barcelona, Spain;3 Department of Electrical;Computer Engineering, Boston University, Boston, MA 02215, USA;4 Human centered AI, Data \& Software, LUXEMBOURG Institute of Science
Comparison of Modern Multilingual Text Embedding Techniques for Hate Speech Detection Task
Department of Information Systems, Faculty of Informatics, Kaunas University of Technology, Kaunas, Lithuania(信息系统系,信息学院,凯纳斯技术大学,凯纳斯,立陶宛);Centre of Information Systems Design Technologies, Faculty of Informatics, Kaunas University of Technology, Kaunas, Lithuania(信息系统设计技术中心,信息学院,凯纳斯技术大学,凯纳斯,立陶宛)
SEA-CLIP-Tiny: Efficient Multilingual Text-Vision Embedding for Southeast Asian Languages
SEACrowd;University of Indonesia(印度尼西亚大学);Cohere Labs Community(Cohere Labs 社区);Technical University of Denmark(丹麦技术大学);Chulalongkorn University(朱拉隆功大学);Vidyasirimedhi Institute of Science and Technology(维迪亚西里梅迪科学技术研究所);AI Singapore(新加坡人工智能)
Department of Computer Science and Engineering, BRAC University(计算机科学与工程系,BRAC大学);Department of Computer Science and Engineering, Ahsanullah University of Science and Technology(计算机科学与工程系,Ahsanullah科技大学)
Evaluating Multilingual Sentence Embeddings for Translation Error Detection:An English--Greek Contrastive Study
Ionian University(爱奥尼亚大学);Laboratory on Digital Libraries and Electronic Publishing(数字图书馆与电子出版实验室);Centre for Literary Translation and Book Policy(文学翻译与图书政策中心)
Multilingual Synopses of Movie Narratives: A Dataset for Vision-Language Story Understanding
College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院)
Cross-modal Speaker Verification and Recognition: A Multilingual Perspective
Istituto Italiano di Tecnologia (IIT)(意大利技术研究院);University of Insubria(因苏布里亚大学);University of Engineering and Technology Taxila(塔克西拉工程技术大学);Information Technology University(信息技术大学)
Visual Word Sense Disambiguation with CLIP through Dual-Channel Text Prompting and Image Augmentations
2 The Bredesen Center for Interdisciplinary Research;3 Department of Electrical Engineering \& Computer Science, University of Tennessee, Knoxville, TN 37916;4 Oak Ridge National Laboratory (ORNL), Oak Ridge, TN 37830
COSMMIC: Comment-Sensitive Multimodal Multilingual Indian Corpus for Summarization and Headline Generation
Department of Computer Science and Engineering, Indian Institute of Technology Patna, India(印度理工学院帕纳分校计算机科学与工程系);Department of Metallurgical and Materials Engineering, National Institute of Technology Tiruchirappalli, India(印度理工学院 Tiruchirappalli 金属与材料工程系);Department of Computer Science and Information Systems, BITS Pilani – Goa Campus, India(比斯·帕尼学院 Goa 分校计算机科学与信息系统系);Department of Computer Science and Engineering, Indian Institute of Information Technology Vadodara, India(印度信息科技学院瓦达拉分校计算机科学与工程系);Department of Computer Science and Engineering, B.M.S. College of Engineering, Bangalore, India(班加罗尔 B.M.S. 工程学院计算机科学与工程系);Université de Toulouse, IRIT UMR 5505 CNRS, France(图卢兹大学 IRIT UMR 5505 CNRS 实验室)
WangLab at MEDIQA-M3G 2024: Multimodal Medical Answer Generation using Large Language Models
University of Toronto(多伦多大学);Vector Institute(向量研究所);Princess Margaret Cancer Centre, University Health Network(大学健康网络玛格丽特公主癌症中心);Lunenfeld-Tanenbaum Research Institute, Sinai Health System(西奈健康系统勒嫩菲尔德-塔南鲍姆研究所);Peter Munk Cardiac Centre, University Health Network(大学健康网络彼得·芒克心脏中心);AI Hub, University Health Network(大学健康网络人工智能中心)
本文介绍了WangLab参加MEDIQA-M3G 2024共享任务的两个方案,分别基于Claude 3 Opus API调用和CLIP风格图像分类,在英语类别中取得第一和第二名,并指出这两种方法在医学视觉问答中具有进一步研究潜力。