This paper presents a novel approach to aligning large language models (LLMs) with individual human preferences, sometimes referred to as Reinforcement Learning from \textit{Personalized} Human Feedback (RLPHF). Given stated preferences along multiple dimensions, such as helpfulness, conciseness, or humor, the goal is to create an LLM without re-training that best adheres to this specification. Starting from specialized expert LLMs, each trained for one such particular preference dimension, we propose a black-box method that merges their outputs on a per-token level. We train a lightweight Preference Control Model (PCM) that dynamically translates the preference description and current context into next-token prediction weights. By combining the expert models' outputs at the token level, our approach dynamically generates text that optimizes the given preference. Empirical tests show that our method matches or surpasses existing preference merging techniques, providing a scalable, efficient alternative to fine-tuning LLMs for individual personalization.
When Personalization Misleads: Understanding and Mitigating Hallucinations in Personalized LLMs
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学首都人工智能学院);AI Lab at Lenovo Research(联想研究院人工智能实验室);School of Artificial Intelligence and Data Science, University of International Business and Economics(国际商务与经济大学人工智能与数据科学学院)
AI PERSONA: Towards Life-long Personalization of LLMs
OPPO AI Center(OPPO AI中心);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Beihang University(北京航空航天大学);Tsinghua University(清华大学);Guangdong University of Technology(广东工业大学)
Personalization of Large Language Models: A Survey
Dartmouth College(达特茅斯学院);Adobe Research(Adobe研究);Stanford University(斯坦福大学);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);Pattern Data;Vanderbilt University(范德比尔特大学);Dolby Research(Dolby研究);University of California San Diego(加州大学圣地亚哥分校);Cisco Research(思科研究);University of Oregon(俄勒冈大学)
Mitigating Over-Personalization in LLMs via Structured Memory
University of Saskatchewan(萨斯喀彻温大学);University of California–Irvine(加州大学欧文分校);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);Obra D. Tompkins High School(奥布拉·D.汤普金斯高中)
Evaluating Personalized Tool-Augmented LLMs from the Perspectives of Personalization and Proactivity
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Rethinking Personalization in Large Language Models at the Token Level
Institute for Artificial Intelligence, Peking University(人工智能研究院,北京大学);State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University(通用人工智能国家重点实验室,北京大学智能科学与技术学院);School of Mathematical Sciences, Peking University(数学科学学院,北京大学)
Reading Between the Prompts: How Stereotypes Shape LLM's Implicit Personalization
Institute for Logic, Language and Computation, University of Amsterdam(逻辑、语言与计算研究所,阿姆斯特丹大学);Center for Language and Cognition, University of Groningen(语言与认知中心,格罗宁根大学)
HARMONI: Multimodal Personalization of Multi-User Human-Robot Interactions with LLMs
Institut Curie, Université Paris-Saclay(巴黎-萨克勒大学Curie研究所);Institute of Intelligent Systems and Robotics (ISIR), Sorbonne University(索邦大学智能系统与机器人研究所);Assistance Publique – Hôpitaux de Paris (AP-HP), Université Paris Cité(巴黎公共医院(AP-HP)与巴黎城市大学)
One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM Personalization
Institute for Natural Language Processing, University of Stuttgart(斯图加特大学自然语言处理研究所);Institute for Logic, Language and Computation, University of Amsterdam(阿姆斯特丹大学逻辑、语言和计算研究所);Weizenbaum Institute(Weizenbaum研究所;慕尼黑机器学习中心;慕尼黑技术大学);Munich Center for Machine Learning;TU Munich