While large multi-modal models (LMM) have shown notable progress in multi-modal tasks, their capabilities in tasks involving dense textual content remains to be fully explored. Dense text, which carries important information, is often found in documents, tables, and product descriptions. Understanding dense text enables us to obtain more accurate information, assisting in making better decisions. To further explore the capabilities of LMM in complex text tasks, we propose the DT-VQA dataset, with 170k question-answer pairs. In this paper, we conduct a comprehensive evaluation of GPT4V, Gemini, and various open-source LMMs on our dataset, revealing their strengths and weaknesses. Furthermore, we evaluate the effectiveness of two strategies for LMM: prompt engineering and downstream fine-tuning. We find that even with automatically labeled training datasets, significant improvements in model performance can be achieved. We hope that this research will promote the study of LMM in dense text tasks. Code will be released at https://github.com/Yuliang-Liu/MultimodalOCR.
PhysicsArena: The First Multimodal Physics Reasoning Benchmark Exploring Variable, Process, and Solution Dimensions
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Beijing Future Brain Education Technology Co., Ltd.(北京未来脑教育科技有限公司);The Hong Kong University of Science and Technology(香港科技大学)
Exploring the Implicit Semantic Ability of Multimodal Large Language Models: A Pilot Study on Entity Set Expansion
Shenzhen International Graduate School Tsinghua University(清华大学深圳国际研究生院);Peng Cheng Laboratory(鹏城实验室);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东省人工智能与数字经济实验室(深圳));Seoul National University(首尔大学)
Imagery as Inquiry: Exploring A Multimodal Dataset for Conversational Recommendation
University of California, San Diego(加利福尼亚大学圣迭戈分校);Institute for Clarity in Documentation(文档清晰度研究所);Inria Paris-Rocquencourt(法国国家信息与自动化研究所巴黎-罗夸恩库尔特中心);Rajiv Gandhi University(拉吉夫·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕尔默研究实验室)
Vrije University of Amsterdam University of Amsterdam Centrum Wiskunde \& Informatic;University of Amsterdam National Institute of Informatics Centrum Wiskunde \& Informatic;Centrum Wiskunde \& Informatic Leiden University University College London;Vrije University of Amsterdam;Centrum Wiskunde \& Informatica Technische Universiteit Delft Guangzhou University;Vrije University of Amsterdam Centrum Wiskunde \& Informatic
From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practice
Kyoto University(京都大学);Georgia Institute of Technology(佐治亚理工学院);National Taiwan Normal University(台湾师范大学);Indiana University(印第安纳大学);Hong Kong University of Science(香港科学大学);The University of Texas at Dallas(德克萨斯大学达拉斯分校);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Cornell University(康奈尔大学);University of Liverpool(利物浦大学);University of Edinburgh(爱丁堡大学);Zhejiang University(浙江大学);Purdue University(Purdue 大学);Emory University, Atlanta, GA, USA(埃默里大学);West China Biomedical Big Data Center, West China Hospital, Sichuan University, Chengdu, China(西京生物大数据中心,四川大学西京医院,成都,中国)
DenseMLLM: Standard Multimodal LLMs for Dense Prediction
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, China(香港科技大学电子与计算机工程系);Tencent, Youtu-Lab, China(腾讯优图实验室)
School of Informatics, Xiamen University, China(厦门大学信息学院);WeChat AI, Tencent Inc, China(腾讯公司微信AI部门);Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China(福建省和台湾非物质文化遗产数字化保护与智能处理重点实验室);Shanghai Artificial Intelligence Laboratory, China(上海人工智能实验室)
Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLMs
School of Data Science, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)数据科学学院);Tencent(腾讯);Renmin University of China(中国人民大学);Johns Hopkins University(约翰霍普金斯大学)
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence(人机混合增强智能国家级重点实验室);Institute of Artificial Intelligence and Robotics(人工智能与机器人研究院);MiLM Plus;Xiaomi Inc(小米公司);Zhongguancun Academy(中关村学院);Beijing, China(北京市)
Exploring the Necessity of Visual Modality in Multimodal Machine Translation using Authentic Datasets
College of Big Data and Internet, Shenzhen Technology University(深圳技术大学大数据与互联网学院);College of Application and Technology, Shenzhen University(深圳大学应用技术学院)
Exploring Human-AI Complementarity in CPS Diagnosis Using Unimodal and Multimodal BERT Models
UCL Knowledge Lab, Institute of Education, University College London, UK(伦敦大学学院教育研究所知识实验室);UCL Centre for Artificial Intelligence, Department of Computer Science, University College London, UK(伦敦大学学院人工智能中心计算机科学系)