Self-Improvement in Multimodal Large Language Models: A Survey
The University of Texas at Dallas(德克萨斯大学达拉斯分校);University of Toronto(多伦多大学);University of Notre Dame(诺特丹大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Shijian Deng, Kai Wang, Tianyu Yang, Harsh Singh, Yapeng Tian
Recent advancements in self-improvement for Large Language Models (LLMs) have efficiently enhanced model capabilities without significantly increasing costs, particularly in terms of human effort. While this area is still relatively young, its extension to the multimodal domain holds immense potential for leveraging diverse data sources and developing more general self-improving models. This survey is the first to provide a comprehensive overview of self-improvement in Multimodal LLMs (MLLMs). We provide a structured overview of the current literature and discuss methods from three perspectives: 1) data collection, 2) data organization, and 3) model optimization, to facilitate the further development of self-improvement in MLLMs. We also include commonly used evaluations and downstream applications. Finally, we conclude by outlining open challenges and future research directions.
A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学);Nanyang Technological University(南洋理工大学);Squirrel Ai Learning(松鼠Ai学习)
Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学);Shanghai AI Laboratory(上海人工智能实验室);Nanyang Technological University(南洋理工大学);Renmin University of China(中国人民大学)
LLMs Meet Multimodal Generation and Editing: A Survey
The Hong Kong University of Science and Technology(香港科技大学);The Chinese University of Hong Kong(香港中文大学);Tsinghua University(清华大学);Tencent AI Lab(腾讯AI实验室)
A Survey of Multimodal Large Language Model from A Data-centric Perspective
Hong Kong University of Science and Technology(香港科技大学);Apple(苹果公司);Peking University(北京大学);Harbin Institute of Technology(哈尔滨工业大学);University of Science and Technology of China(中国科学技术大学)
Bridging Text and Molecule: A Survey on Multimodal Frameworks for Molecule
Center for Research on Intelligent Perception and Computing (CRIPAC)(智能感知与计算研究中心);State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS)(多模态人工智能系统国家重点实验室);Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
School of Automation Science and Engineering, Xi’an Jiaotong University(自动化科学与工程学院,西安交通大学);Interdisciplinary Graduate Programme, Nanyang Technological University(跨学科研究生项目,南洋理工大学);Lenovo Research, Lenovo(联想研究院,联想);School of Computer Science and Technology, Xi’an Jiaotong University(计算机科学与技术学院,西安交通大学);College of Computing and Data Science, Nanyang Technological University(计算与数据科学学院,南洋理工大学)
MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
Nanjing University(南京大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Science and Technology of China(中国科学技术大学);Nanyang Technological University(南洋理工大学)
A Survey of Generative Categories and Techniques in Multimodal Generative Models
School of Architecture, Technology and Engineering, University of Brighton, Lewes Road, BN2 4GJ(建筑、科技与工程学院,布里顿大学,勒斯路,BN2 4GJ);University of Khorfakkan, Sharjah(柯法克坎大学,沙迦)
How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model
College of Computer, National University of Defense Technology(国防科技大学计算机学院);School of Computer and Information Engineering, Hefei University of Technology(合肥工业大学计算机与信息工程学院)
Personalized Multimodal Large Language Models: A Survey
University of California, San Diego(加利福尼亚大学圣地亚哥分校);University of Rochester(罗切斯特大学);Dartmouth College(达特茅斯学院);Adobe Research(奥多比研究院);University of Massachusetts at Amherst(马萨诸塞大学阿默斯特分校);University of Southern California(南加利福尼亚大学);Virginia Tech(弗吉尼亚理工大学);Cisco Research(思科研究院);University of Oregon(俄勒冈大学);Meta AI;University of Arizona(亚利桑那大学);University of Maryland(马里兰大学)
From Models to Systems: A Comprehensive Survey of Efficient Multimodal Learning
University of Pittsburgh(匹兹堡大学);Stanford University(斯坦福大学);UCSD(加州大学圣地亚哥分校);University of Arizona(亚利桑那大学);Georgia Tech(佐治亚理工学院);Carnegie Mellon University(卡内基梅隆大学);UNC Chapel Hill(北卡罗来纳大学教堂山分校);UC Merced(加州大学默塞德分校);New York University(纽约大学)
From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models
School of Computer Science, Sichuan University(四川大学计算机学院);School of Electronic and Computer Engineering, Peking University Shenzhen Graduate School(北京大学深圳研究生院电子与计算机工程学院);Institute of Artificial Intelligence (TeleAI), China Telecom and Northwestern Polytechnical University(中国电信与西北工业大学人工智能研究院(TeleAI))
Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey
Peking University(北京大学);Beihang Univerisy(北京航空航天大学);University of Hongkong(香港大学);Shenzhen Institute of Advanced Technology, China Academy of Sciences(中国科学院深圳先进技术研究院);Tsinghua University(清华大学);M-A-P;The Chinese University of Hong Kong(香港中文大学);Alibaba Group(阿里巴巴集团);University of Cambridge(剑桥大学);Microsoft Research(微软研究院);UIUC(伊利诺伊大学厄巴纳-香槟分校);Humanify Inc.(Humanify公司);Zhejiang University(浙江大学)
Multimodal Large Language Models for Medicine: A Comprehensive Survey
School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院);School of Computer Science, Peking University(北京大学计算机科学学院)
A Survey on Agentic Multimodal Large Language Models
Nanyang Technological University, Singapore(南洋理工大学,新加坡);Chinese University of Hong Kong, Shenzhen, China(香港中文大学(深圳));Shenzhen Research Institute of Big Data, China(大数据研究 institute);Sun Yat-sen University, China(中山大学);City University of Hong Kong, China(香港城市大学);Communication University of China, China(通信大学)
Language and Multimodal Models in Sports: A Survey of Datasets and Applications
University of California, Irvine(加利福尼亚大学欧文分校);George Mason University(乔治梅森大学);Meta Platforms, Inc.(元平台公司);Stanford University(斯坦福大学);University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);Rice University(莱斯大学)
Multimodal Emotion Recognition in Conversations: A Survey of Methods, Trends, Challenges and Prospects
Guangdong Provincial Key Laboratory of Quantum Engineering and Quantum Materials(广东省量子工程与量子材料重点实验室);School of Electronic Science and Engineering (School of Microelectronics), South China Normal University(华南师范大学电子科学学院(微电子学院));North Carolina Central University(北卡罗来纳中央大学);Georgia Institute of Technology(佐治亚理工学院);Columbia University(哥伦比亚大学);LMU Munich & Munich Center for Machine Learning(慕尼黑大学及慕尼黑机器学习中心)
Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
Computer Engineering Department, Sharif University of Technology(沙斐大学计算机工程系);College of Interdisciplinary Science and Technology, University of Tehran(德黑兰大学跨学科科学与技术学院);Computer Engineering Department, K.N. Toosi University of Technology(技术学院计算机工程系);Qatar Computing Research Institute(卡塔尔计算研究院)
Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation
The University of Hong Kong(香港大学);Tencent AI Lab(腾讯人工智能实验室);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);East China Normal University(华东师范大学)
Jailbreak Attacks and Defenses against Multimodal Generative Models: A Survey
School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院);School of Computer Science, University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校计算机科学学院);State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(中国科学院自动化研究所多模态人工智能系统国家重点实验室);New Laboratory of Pattern Recognition, CASIA(中国科学院自动化研究所新模式识别实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)