我们提出了一个新颖的框架,通过利用经过微调的多模态语言模型(Multimodal Language Models,MLM)来过滤图文数据。通过整合MLM的最新进展,我们的方法优于主流过滤方法(例如 CLIPScore)。我们设计了四个不同但互补的指标,以全面衡量图文数据的质量。我们还建立了一条新的流程,用于构建高质量指令数据,从而将MLM微调为数据过滤器。与CLIPScore相比,我们的MLM过滤器能够产生更精确、更全面的分数,直接提升被过滤数据的质量,并提高预训练模型的性能。在流行的基础模型(即CLIP和BLIP2)以及多种下游任务上,我们相较CLIPScore取得了显著改进。我们的MLM过滤器能够泛化到不同模型和任务,并可作为CLIPScore的即插即用替代方案。本文还提供了额外的消融研究,以验证我们针对MLM过滤器的设计选择。
英文摘要:
We propose a novel framework for filtering image-text data by leveraging fine-tuned Multimodal Language Models (MLMs). Our approach outperforms predominant filtering methods (e.g., CLIPScore) via integrating the recent advances in MLMs. We design four distinct yet complementary metrics to holistically measure the quality of image-text data. A new pipeline is established to construct high-quality instruction data for fine-tuning MLMs as data filters. Comparing with CLIPScore, our MLM filters produce more precise and comprehensive scores that directly improve the quality of filtered data and boost the performance of pre-trained models. We achieve significant improvements over CLIPScore on popular foundation models (i.e., CLIP and BLIP2) and various downstream tasks. Our MLM filter can generalize to different models and tasks, and be used as a drop-in replacement for CLIPScore. An additional ablation study is provided to verify our design choices for the MLM filter.
Maya: An Instruction Finetuned Multilingual Multimodal Model
Cohere For AI Community(Cohere 人工智能社区);Indiana University Bloomington(印第安纳大学伯明顿分校);Imperial College London(伦敦帝国学院);Georgia Institute of Technology(佐治亚理工学院);The Alan Turing Institute(艾伦·图灵研究所);Bangladesh University of Engineering and Technology(孟加拉工程技术大学);University of Pennsylvania(宾夕法尼亚大学);IIT Bombay(印度理工学院孟买分校);TU Darmstadt(达姆施塔特工业大学);IIT Dhanbad(印度理工学院丹巴德分校);MBZUAI(穆罕默德·本·扎耶德人工智能大学);Cisco Meraki(思科 Meraki);Articul8 AI(Articul8 人工智能公司);Capital One(第一资本金融公司)
Person-Centric Annotations of LAION-400M: Auditing Bias and Its Transfer to Models
Technical University of Munich, Munich Center for Machine Learning (MCML), MDSI(慕尼黑工业大学,慕尼黑机器学习中心(MCML),MDSI);LTCI, Télécom Paris, Institut Polytechnique de Paris, France(法国巴黎理工学院,巴黎电信学院,LTCI);Helmholtz Munich(亥姆霍兹慕尼黑中心);University of California, Berkeley(加州大学伯克利分校)
A Robust Multimodal Remote Sensing Image Registration Method and System Using Steerable Filters with First- and Second-order Gradients
Southwest Jiaotong University(西南交通大学);Beijing Institute of Technology(北京理工学院);State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing(测绘遥感信息工程国家重点实验室)
Local-sensitive connectivity filter (ls-cf): A post-processing unsupervised improvement of the frangi, hessian and vesselness filters for multimodal vessel segmentation
Department of Academic Informatics (DAINF), Universidade Tecnologica Federal do Parana (UTFPR)(学术信息系(DAINF),技术联邦大学帕托布拉诺分校(UTFPR));Graduate Program of Sciences Applied to Health Products, Universidade Federal Fluminense (UFF)(健康产品应用科学研究生项目,联邦大学弗洛里塞分校(UFF));Institute of Technological Sciences (ICT), Universidade Federal de Itajuba (UNIFEI)(技术科学研究所,联邦大学伊塔比亚分校(UNIFEI));Department of Electrical Engineering and Computer Science, Khalifa University of Science and Technology(电气工程与计算机科学系,卡利法科学技术大学)
Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所MAIS);Alibaba Cloud Computing(阿里巴巴云计算)
Visual Question Decomposition on Multimodal Large Language Models
Munich Center for Machine Learning(慕尼黑机器学习中心);Technical University of Munich(慕尼黑工业大学);Amazon Web Services(亚马逊云计算服务);LMU Munich(慕尼黑大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学);University of Oxford(牛津大学)
Generative artificial intelligence in ophthalmology: multimodal retinal images for the diagnosis of Alzheimer's disease with convolutional neural networks
Contrast Sensitivity in Multimodal Large Language Models: A Psychophysics-Inspired Evaluation
Image Processing Lab, Universitat de València, Spain(瓦伦西亚大学图像处理实验室);Computer Vision Center, Spain(西班牙计算机视觉中心);Universitat Autònoma de Barcelona, Spain(巴塞罗那自治大学);Center for Biomaterials and Tissue Engineering, Universitat Politecnica de Valencia, Spain(瓦伦西亚理工大学生物材料与组织工程中心)
Universally Unfiltered and Unseen:Input-Agnostic Multimodal Jailbreaks against Text-to-Image Model Safeguards
Information Engineering University Zhengzhou China;School of Computer Science, \ University Wuhan China;Xi’an Jiaotong University Xi’an China;Wuhan University Wuhan China;Information Engineering University;School of Computer Science, \ University;Xi’an Jiaotong University;Wuhan University
CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation
The Hong Kong University of Science and Technology(香港科技大学);Wuhan University(武汉大学);AI Chip Center for Emerging Smart Systems(新兴智能系统人工智能芯片中心);Kuaishou Technology(快手科技)
A Foundational Multimodal Vision Language AI Assistant for Human Pathology
Brigham and Women’s Hospital(布莱根妇女医院);Harvard Medical School(哈佛医学院);Massachusetts General Hospital(麻省总医院);Broad Institute of Harvard and MIT(哈佛-麻省理工博德研究所);Dana-Farber Cancer Institute(达纳-法伯癌症研究所);Massachusetts Institute of Technology (MIT)(麻省理工学院);Harvard John A. Paulson School of Engineering And Applied Sciences(哈佛大学约翰·A·保尔森工程与应用科学学院);Harvard University(哈佛大学);Wexner Medical Center, Ohio State University(俄亥俄州立大学韦克斯纳医学中心);Ohio State University(俄亥俄州立大学);Harvard Data Science Initiative(哈佛大学数据科学计划)
Multimodal Pragmatic Jailbreak on Text-to-image Models
LMU Munich, Germany(慕尼黑大学);Munich Center for Machine Learning, Germany(慕尼黑机器学习中心);Saarland University, Germany(萨尔兰大学);Max Planck Institute for Informatics, Germany(马克斯·普朗克信息研究所);Cornell University, USA(康奈尔大学);University of Oxford, UK(牛津大学)