A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
School of Computer Science, Northwestern Polytechnical University(西北工业大学计算机学院);School of Automation, Northwestern Polytechnical University(西北工业大学自动化学院);Institute of Medical Research, Northwestern Polytechnical University(西北工业大学医学研究院);School of Computing, The University of Georgia(佐治亚大学计算学院);School of Physics and Information Technology, Shaanxi Normal University(陕西师范大学物理与信息技术学院)
In an era defined by the explosive growth of data and rapid technological advancements, Multimodal Large Language Models (MLLMs) stand at the forefront of artificial intelligence (AI) systems. Designed to seamlessly integrate diverse data types-including text, images, videos, audio, and physiological sequences-MLLMs address the complexities of real-world applications far beyond the capabilities of single-modality systems. In this paper, we systematically sort out the applications of MLLM in multimodal tasks such as natural language, vision, and audio. We also provide a comparative analysis of the focus of different MLLMs in the tasks, and provide insights into the shortcomings of current MLLMs, and suggest potential directions for future research. Through these discussions, this paper hopes to provide valuable insights for the further development and application of MLLM.
MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routing
School of Artificial Intelligence, Nanjing University(人工智能学院,南京大学);National Key Laboratory for Novel Software Technology, Nanjing University(新型软件技术国家实验室,南京大学)
Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency
University of Guelph(圭尔夫大学);University of North Texas(北德克萨斯大学);Iowa State University(爱荷华州立大学);Zayed University(扎耶德大学);University of Texas at Dallas(德克萨斯大学达拉斯分校);National University of Computer & Emerging Sciences(国立计算机与新兴科学大学)
Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark
Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Pattern Recognition Center, WeChat AI, Tencent Inc, China(腾讯人工智能研究院);Kennesaw State University(凯斯西储大学)
Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
Computer Engineering Department, Sharif University of Technology(沙斐大学计算机工程系);College of Interdisciplinary Science and Technology, University of Tehran(德黑兰大学跨学科科学与技术学院);Computer Engineering Department, K.N. Toosi University of Technology(技术学院计算机工程系);Qatar Computing Research Institute(卡塔尔计算研究院)
OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks
Chinese University of Hong Kong(香港中文大学);Sun Yat-sen Memorial Hospital, Sun Yat-sen University(中山大学孙逸仙纪念医院);Southern Medical University(南方医科大学);Zhongshan Hospital, Fudan University(复旦大学中山医院);Department of Neurosurgery, Prince of Wales Hospital(威尔士王室医院神经外科部)
The Paradigm Shift: A Comprehensive Survey on Large Vision Language Models for Multimodal Fake News Detection
College of Computer and Mathematics, Central South University of Forestry and Technology(计算机与数学学院,中央南大学林业科技学院);College of Computer Science and Electronic Engineering, Hunan University(计算机科学与电子工程学院,湖南大学);College of Economics and Management, Central South University of Forestry and Technology(经济管理学院,中央南大学林业科技学院);Department of Computer Science, State University of New York(计算机科学系,纽约州立大学)
LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating
MAIS, Institute of Automation, Chinese Academy of Sciences(人工智能研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学);Taobao & Tmall Group of Alibaba(阿里巴巴淘宝与天猫集团);Zhongguancun Academy, Beijing(中关村学院,北京)
Advancing Object Detection in Transportation with Multimodal Large Language Models (MLLMs): A Comprehensive Review and Empirical Testing
Arab American University(阿拉伯美国大学);Columbia University(哥伦比亚大学);Budapest University of Technology and Economics(布达佩斯技术与经济大学);Al-Ahliyya Amman University(阿赫利耶阿曼大学);Queensland University of Technology(昆士兰科技大学)
MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding
Beijing Academy of Artificial Intelligence (BAAI), China(北京人工智能研究院);Institute of Information Engineering, Chinese Academy of Sciences, China(信息工程研究所);Beijing University of Technology, China(北京理工大学)
Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey
Peking University(北京大学);Beihang Univerisy(北京航空航天大学);University of Hongkong(香港大学);Shenzhen Institute of Advanced Technology, China Academy of Sciences(中国科学院深圳先进技术研究院);Tsinghua University(清华大学);M-A-P;The Chinese University of Hong Kong(香港中文大学);Alibaba Group(阿里巴巴集团);University of Cambridge(剑桥大学);Microsoft Research(微软研究院);UIUC(伊利诺伊大学厄巴纳-香槟分校);Humanify Inc.(Humanify公司);Zhejiang University(浙江大学)
MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
Nanjing University(南京大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Science and Technology of China(中国科学技术大学);Nanyang Technological University(南洋理工大学)
INS-MMBench: A Comprehensive Benchmark for Evaluating LVLMs' Performance in Insurance
School of Computer Science Fudan University(复旦大学计算机科学学院);Department of Computer Science University of Rochester(罗切斯特大学计算机科学系);School of Economics Fudan University(复旦大学经济学院)
From Models to Systems: A Comprehensive Survey of Efficient Multimodal Learning
University of Pittsburgh(匹兹堡大学);Stanford University(斯坦福大学);UCSD(加州大学圣地亚哥分校);University of Arizona(亚利桑那大学);Georgia Tech(佐治亚理工学院);Carnegie Mellon University(卡内基梅隆大学);UNC Chapel Hill(北卡罗来纳大学教堂山分校);UC Merced(加州大学默塞德分校);New York University(纽约大学)
MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agents
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);College of Information and Electrical Engineering, China Agricultural University(中国农业大学信息与电气工程学院)
A review of deep learning-based information fusion techniques for multimodal medical image classification
Inserm(法国国家健康与医学研究院);University of Western Brittany(西布列塔尼大学);IMT Atlantique(大西洋电信学院);Sorbonne University(索邦大学);Lariboisière Hospital(拉里布瓦西耶尔医院);AP-HP(巴黎公共医疗集团);Paris Cité University(巴黎城市大学);CHRU Brest(布雷斯特大学中心医院);LaTIM UMR 1101(LaTIM联合研究单位1101)
Multimodal Fusion on Low-quality Data: A Comprehensive Survey
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局高性能计算研究所);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);College of Computer Science, Sichuan University(四川大学计算机学院);School of Computer Science and Technology, Xidian University(西安电子科技大学计算机科学与技术学院);School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳)计算机科学与技术学院)
Multimodal Clinical Benchmark for Emergency Care (MC-BEC): A Comprehensive Benchmark for Evaluating Foundation Models in Emergency Medicine
Harvard Medical School(哈佛医学院);Stanford University(斯坦福大学);Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University(哈佛大学约翰·A·保尔森工程与应用科学学院)
A Review on Explainability in Multimodal Deep Neural Nets
Symbiosis Institute of Technology (SIT)(共生工程技术学院(SIT));Symbiosis International (Deemed University)(共生国际大学);Symbiosis Centre for Applied Artificial Intelligence(共生应用人工智能中心)
Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Survey
Saw Swee Hock School of Public Health, National University of Singapore(新加坡国立大学公共健康学院);School of Computer Science, Beijing University of Posts and Telecommunications(北京邮电大学计算机学院);School of Automation, Northwestern Polytechnical University(西北工业大学自动化学院);School of Computer Science and Technology, Xi’an Jiaotong University(西安交通大学计算机科学与技术学院)
Navigating the landscape of multimodal AI in medicine: a scoping review on technical challenges and clinical applications
Radboud University Medical Center(拉德堡德大学医学中心);Oncode Institute(Oncode研究所);Polytechnic University of Turin(都灵理工大学);University of Turin(都灵大学);Erasmus University Medical Centre(伊拉斯姆斯大学医学中心)
FMMD: A multimodal multidisciplinary dataset of open peer reviews from F1000Research
School of Computer Science and Engineering, Guangzhou Institute of Science and Technology(计算机科学与工程学院,广州科学与技术研究所);College of Foreign Languages and Cultures, Xiamen University(外语学院,厦门大学);Science and Education Evaluation Lab, Guangzhou Institute of Science and Technology(科学与教育评估实验室,广州科学与技术研究所)
Review of multimodal machine learning approaches in healthcare
Oxford Internet Institute, University of Oxford(牛津大学互联网研究所);Institute for People-Centred AI, University of Surrey(萨里大学以人为本人工智能研究所);Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London(伦敦大学学院惠康/英国工程与物理科学研究委员会介入与外科科学中心)
Multimodal Federated Learning in Healthcare: a Review
West Virginia University(西弗吉尼亚大学);Tribhuvan University(特里布万大学);University at Albany(奥尔巴尼大学);University of Pittsburgh(匹兹堡大学);University of Aberdeen(阿伯丁大学)
Multimodal Detection of Fake Reviews using BERT and ResNet-50
College of Business Administration Kansas State University Manhattan, USA(商学院学院 华盛顿州立大学 曼哈顿 美国);Department of Computer Science Kansas State University Manhattan, USA(计算机科学系 华盛顿州立大学 曼哈顿 美国)
针对单模态检测无法捕捉跨模态语义不一致的问题,提出集成 BERT 文本与 ResNet-50 图像特征的多模态虚假评论检测框架,在 21,142 张图像数据集上 F1 达 0.934,有效识别图文不一致的虚假评论。