Personalized Multimodal Large Language Models: A Survey
University of California, San Diego（加利福尼亚大学圣地亚哥分校）；University of Rochester（罗切斯特大学）；Dartmouth College（达特茅斯学院）；Adobe Research（奥多比研究院）；University of Massachusetts at Amherst（马萨诸塞大学阿默斯特分校）；University of Southern California（南加利福尼亚大学）；Virginia Tech（弗吉尼亚理工大学）；Cisco Research（思科研究院）；University of Oregon（俄勒冈大学）；Meta AI；University of Arizona（亚利桑那大学）；University of Maryland（马里兰大学）
Junda Wu, Hanjia Lyu, Yu Xia, Zhehao Zhang, Joe Barrow, Ishita Kumar, Mehrnoosh Mirtaheri, Hongjie Chen, Ryan A. Rossi, Franck Dernoncourt, Tong Yu, Ruiyi Zhang… 展开作者
Junda Wu, Hanjia Lyu, Yu Xia, Zhehao Zhang, Joe Barrow, Ishita Kumar, Mehrnoosh Mirtaheri, Hongjie Chen, Ryan A. Rossi, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, Jiuxiang Gu, Nesreen K. Ahmed, Yu Wang, Xiang Chen, Hanieh Deilamsalehy, Namyong Park, Sungchul Kim, Huanrui Yang, Subrata Mitra, Zhengmian Hu, Nedim Lipka, Dang Nguyen, Yue Zhao, Jiebo Luo, Julian McAuley
Multimodal Large Language Models (MLLMs) have become increasingly important due to their state-of-the-art performance and ability to integrate multiple data modalities, such as text, images, and audio, to perform complex tasks with high accuracy. This paper presents a comprehensive survey on personalized multimodal large language models, focusing on their architecture, training methods, and applications. We propose an intuitive taxonomy for categorizing the techniques used to personalize MLLMs to individual users, and discuss the techniques accordingly. Furthermore, we discuss how such techniques can be combined or adapted when appropriate, highlighting their advantages and underlying rationale. We also provide a succinct summary of personalization tasks investigated in existing research, along with the evaluation metrics commonly used. Additionally, we summarize the datasets that are useful for benchmarking personalized MLLMs. Finally, we outline critical open challenges. This survey aims to serve as a valuable resource for researchers and practitioners seeking to understand and advance the development of personalized multimodal large language models.
Enhancing Zero-shot Personalized Image Aesthetics Assessment with Profile-aware Multimodal LLM
Mashang Consumer Finance Co., Ltd.（马莎消费金融有限公司）；Xi'an Jiaotong-Liverpool University（西安交通大学利物浦大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
Discrete Prompt Tuning via Recursive Utilization of Black-box Multimodal Large Language Model for Personalized Visual Emotion Recognition
Graduate School of Information Science（信息科学研究生学校）；Technology, Hokkaido University, Sapporo 060-0814, Japan（技术，北海道大学，札幌060-0814，日本）；Office of Institutional Research, Hokkaido University, Sapporo 060-0808, Japan（机构研究办公室，北海道大学，札幌060-0808，日本）；Faculty of Information Science（信息科学学院）
ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistant
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；School of Engineering, Westlake University（西湖大学工程学院）；University of Minnesota – Twin Cities（明尼苏达大学双城分校）；University of Toronto（多伦多大学）
A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
University of California, San Diego（加利福尼亚大学圣地亚哥分校）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Adobe Research（奥多比研究院）；The Ohio State University（俄亥俄州立大学）；Allen Institute for AI（艾伦人工智能研究所）
A Survey of Personalized Large Language Models: Progress and Future Directions
The Chinese University of Hong Kong（香港中文大学）；Huawei Technologies Co., Ltd（华为技术有限公司）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；National University of Singapore（新加坡国立大学）
PPU-Bench:Real World Benchmark for Personalized Partial Unlearning in Vision Language Models
Harbin Institute of Technology（哈尔滨工业大学）；Pengcheng Laboratory（鹏城实验室）；The Hong Kong Polytechnic University（香港理工大学）；Sichuan University（四川大学）；Zhejiang Normal University（浙江师范大学）
A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications
Ant Group（蚂蚁集团）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）
A Survey on Foundation Models for Personalized Federated Intelligence
School of Computing, Kyung Hee University（韩国庆熙大学计算机学院）；Noakhali Science and Technology University（诺阿克利科学与技术大学）；Korea Advanced Institute of Science and Technology（韩国科学技术院）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）
Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
Computer Engineering Department, Sharif University of Technology（沙斐大学计算机工程系）；College of Interdisciplinary Science and Technology, University of Tehran（德黑兰大学跨学科科学与技术学院）；Computer Engineering Department, K.N. Toosi University of Technology（技术学院计算机工程系）；Qatar Computing Research Institute（卡塔尔计算研究院）
Large AI Model Empowered Multimodal Semantic Communications
Hunan Normal University（湖南师范大学）；Hunan University of Technology and Business（湖南工商大学）；Brunel University（布鲁内尔大学）；Nanjing University（南京大学）；Southeast University（东南大学）
LLMs Meet Multimodal Generation and Editing: A Survey
The Hong Kong University of Science and Technology（香港科技大学）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）；Tencent AI Lab（腾讯AI实验室）
Personalization of Large Language Models: A Survey
Dartmouth College（达特茅斯学院）；Adobe Research（Adobe研究）；Stanford University（斯坦福大学）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Pattern Data；Vanderbilt University（范德比尔特大学）；Dolby Research（Dolby研究）；University of California San Diego（加州大学圣地亚哥分校）；Cisco Research（思科研究）；University of Oregon（俄勒冈大学）
TaleForge: Interactive Multimodal System for Personalized Story Creation
1 University of Science, VNU-HCM, Ho Chi Minh City, Vietnam；2 Vietnam National University, Ho Chi Minh City, Vietnam；3 Department of Computer Science \ of Dayton Ohio, United States
