Compared to natural images, understanding scientific figures is particularly hard for machines. However, there is a valuable source of information in scientific literature that until now has remained untapped: the correspondence between a figure and its caption. In this paper we investigate what can be learnt by looking at a large number of figures and reading their captions, and introduce a figure-caption correspondence learning task that makes use of our observations. Training visual and language networks without supervision other than pairs of unconstrained figures and captions is shown to successfully solve this task. We also show that transferring lexical and semantic knowledge from a knowledge graph significantly enriches the resulting features. Finally, we demonstrate the positive impact of such features in other tasks involving scientific text and figures, like multi-modal classification and machine comprehension for question answering, outperforming supervised baselines and ad-hoc approaches.
MSEarth: A Multimodal Scientific Dataset and Benchmark for Phenomena Uncovering in Earth Science
The Hong Kong Polytechnic University(香港理工大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学)
Do Large Multimodal Models Solve Caption Generation for Scientific Figures? Lessons Learned from SciCap Challenge 2023
Pennsylvania State University(宾夕法尼亚州立大学);National Tsing Hua University(国立清华大学);AllSci(奥尔赛公司);MetaMetrics Inc.(美塔麦特里克公司);Adobe Research(奥多比研究院);Institute of Information Science, Academia Sinica(中央研究院资讯科学研究所)
Generating Accurate and Detailed Captions for High-Resolution Images
Department of Artificial Intelligence, University of Seoul(首尔大学人工智能系);Department of Computer Science and Engineering, POSTECH(POSTECH计算机科学与工程系);Department of Applied Statistics, Yonsei University(延世大学应用统计系)
MatMMExtract: An Open-Source Pipeline for Panel-Level Extraction of Grounded Image-Text Pairs from Materials Science Literature
Mehta Family School of Data Science and Artificial Intelligence(梅塔家庭数据科学与人工智能学院);Indian Institute of Technology Roorkee(印度理工学院罗奥克学院);Department of Metallurgical and Materials Engineering(冶金与材料工程系)
From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France(斯特拉斯堡大学、法国国家科学研究中心、法国国家医学研究院、ICube、UMR7357、斯特拉斯堡,法国);Technical University of Munich(慕尼黑技术大学);Stanford University(斯坦福大学);DSAI, The Hong Kong Polytechnic University(香港理工大学DSAI实验室);University of British Columbia(不列颠哥伦比亚大学);Munich Center for Machine Learning(慕尼黑机器学习中心);IHU Strasbourg, Strasbourg(斯特拉斯堡IHU医院,斯特拉斯堡);Vector Institute for AI(人工智能向量研究所);Yale University(耶鲁大学)
Department of Biomedical Engineering, Faculty of Engineering, Mahidol University(生物医学工程系,工程学院,玛希多大学);Faculty of Medicine Ramathibodi Hospital, Mahidol University(拉玛蒂波迪医院医学院,玛希多大学)
How Do VLMs Behave When Blind or Misled? Behavioral Evaluation of VLMs on Scientific Figures
University of Aberdeen(阿伯丁大学);International Institute of Information Technology Hyderabad(海德拉巴国际信息技术学院);University of Technology Nuremberg(纽伦堡工业大学);University of Southern California(南加利福尼亚大学)
Beyond Text and Tables: Vision-Language Model Integration in ComProScanner for Extracting Materials Data from Scientific Figures with High Accuracy
Energy, Materials and Environment Research Centre, London South Bank University, London SE1 0AA, UK(能源、材料与环境研究中心,伦敦南银行大学);School of Engineering and Design, London South Bank University, London SE1 0AA, UK(工程与设计学院,伦敦南银行大学);Bioscience and Bioengineering Research Centre, London South Bank University, London SE1 0AA, UK(生物科学与生物工程研究中心,伦敦南银行大学);Department of Physics, Kings College London, London WC2R 2LS, UK(物理系,伦敦国王学院)
Can Vision-Language Models Understand and Interpret Dynamic Gestures from Pedestrians? Pilot Datasets and Exploration Towards Instructive Nonverbal Commands for Cooperative Autonomous Vehicles
University of California Merced(加州大学默塞德分校);Aalborg University(奥尔堡大学)