We introduce Lumos, the first end-to-end multimodal question-answering system with text understanding capabilities. At the core of Lumos is a Scene Text Recognition (STR) component that extracts text from first person point-of-view images, the output of which is used to augment input to a Multimodal Large Language Model (MM-LLM). While building Lumos, we encountered numerous challenges related to STR quality, overall latency, and model inference. In this paper, we delve into those challenges, and discuss the system architecture, design choices, and modeling techniques employed to overcome these obstacles. We also provide a comprehensive evaluation for each component, showcasing high quality and efficiency.
Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
School of Automation Science and Engineering, Xi’an Jiaotong University（自动化科学与工程学院，西安交通大学）；Interdisciplinary Graduate Programme, Nanyang Technological University（跨学科研究生项目，南洋理工大学）；Lenovo Research, Lenovo（联想研究院，联想）；School of Computer Science and Technology, Xi’an Jiaotong University（计算机科学与技术学院，西安交通大学）；College of Computing and Data Science, Nanyang Technological University（计算与数据科学学院，南洋理工大学）
VFA: Empowering Multilingual MLLMs via Vision-Free Adaptation
Southern University of Science and Technology（南方科技大学）；Microsoft Research Asia（微软亚洲研究院）；Shanghai University of Finance and Economics（上海财经大学）；Peking University（北京大学）
MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Alibaba Group（阿里巴巴集团）；Tongji University（同济大学）；The University of Sydney（悉尼大学）
ForgerySleuth: Empowering Multimodal Large Language Models for Image Manipulation Detection
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University（上海智能信息处理关键实验室，复旦大学计算机学院）；Shanghai Collaborative Innovation Center of Intelligent Visual Computing（上海智能视觉计算协同创新中心）
ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability
Nankai University（南开大学）；Shanghai Innovation Institute（上海创新研究院）；University of Science and Technology of China（中国科学技术大学）；Wuhan University（武汉大学）；Shanghai AI Laboratory（上海人工智能实验室）
CVSearch: Empowering Multimodal LLMs with Cognitive Visual Search for High-Resolution Image Perception
Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学（深圳））；Peng Cheng Laboratory, Shenzhen, China（鹏城实验室）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China（深圳先进技术研究院）
Cross-Linguistic Persona-Driven Data Synthesis for Robust Multimodal Cognitive Decline Detection
Engineering Research Center of Intelligent Theranostics Technology and Instruments, Ministry of Education, School of Biomedical Engineering and Informatics, Nanjing Medical University（智能诊疗技术与仪器工程研究中心、教育部、生物医学工程与信息学院、南京医科大学）；Institute of Biomedical Engineering, University of Oxford（生物医学工程研究所、牛津大学）；Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR)（高性能计算研究所、科技研究局（A*STAR））；Department of Neurology, National Neuroscience Institute, Singapore, 308433, Singapore（神经病学部、新加坡国家神经科学研究所）；Duke-NUS Medical School, Singapore, 169857, Singapore（新加坡国立大学医学院）
Merlin:Empowering Multimodal LLMs with Foresight Minds
Huazhong University of Science and Technology（华中科技大学）；ShanghaiTech University（上海科技大学）；Beijing Institute of Technology（北京理工大学）；University of Chinese Academy of Sciences（中国科学院大学）；MEGVII Technology（旷视科技）
InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling
Shanghai AI Laboratory（上海人工智能实验室）；Nanjing University（南京大学）；Shanghai Innovation Institute（上海创新研究院）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）
GeoLink: Empowering Remote Sensing Foundation Model with OpenStreetMap Data
School of Earth and Space Sciences, Peking University（地球与空间科学学院，北京大学）；College of Urban and Environmental Sciences, Peking University（城市与环境科学学院，北京大学）；State Key Laboratory of Multimodal Artificial Intelligence Systems Institute of Automation, CAS（多模态人工智能系统国家重点实验室，中国科学院自动化研究所）；Intelligent Maintenance and Operations Systems Lab（智能维护与运营系统实验室）
SpiroLLM: Finetuning Pretrained LLMs to Understand Spirogram Time Series with Clinical Validation in COPD Reporting
Guangzhou Institute of Technology, Xidian University, Xi’an, China（广州科技研究院，西安电子科技大学，中国）；Department of Computer Science, Tianjin University of Technology, Tianjin, China（天津理工大学计算机学院，天津，中国）；Department of Respiratory, The Second Hospital of Tianjin Medical University, China（天津医科大学第二医院呼吸科，中国）；College of Pulmonary and Critical Care Medicine, Chinese PLA General Hospital, Beijing, China（中国人民解放军总医院呼吸与危重症医学科，北京，中国）；HeartVoice Medical Technology, Hefei, China（合肥心声医疗技术有限公司，中国）；School of Life Science and Technology, Xidian University, Xi’an, China（西安电子科技大学生命科学与技术学院，中国）；National Institute of Health Data Science, Peking University, Beijing, China（北京大学国家健康数据科学研究院，北京，中国）；Institute for Artificial Intelligence, Peking University, Beijing, China（北京大学人工智能研究院，北京，中国）
SnapBench: Benchmarking Snap-and-Ask Multimodal Retrieval for Mobile Interactions
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Tencent Yuanbao（腾讯元宝）；Tsinghua University（清华大学）；ARC Lab, Tencent（腾讯ARC实验室）；The University of Hong Kong（香港大学）
Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answering
organization= College of Electrical；Information Engineering,Hunan University , city= Changsha , postcode= 410082 , state= Hunan , country= China；organization= School of Computing \& Information Systems ,The University of Melbourne , city= Melbourne , postcode= VIC 3053 , state= VIC , country= Australia；organization= School of Computer Science,The University of Sydney , city= Sydney , postcode= NSW 2006 , state= NSW , country= Australia；organization= School of Artificial Intelligence ,Anhui University , city= Hefei , postcode= 230601 , state= Anhui , country= China
