Aligning large language models (LLMs) with diverse and multifaceted user preferences is a fundamental challenge in personalized AI systems. Existing multi-objective alignment methods either rely on costly training or require pre-trained reward models for each preference, making it difficult for them to adapt to evolving preferences. Prompt-based personalization offers a training-free alternative, but prompting alone often provides limited steerability, as LLMs may overemphasize or overlook certain preferences and fail to give users reliable control over the relative importance of different objectives when conflicts arise, leading to suboptimal alignment. In this paper, we introduce MATO, a training-free framework for Multi-objective personalized Alignment with Test-time Optimization. MATO formulates personalization as a test-time optimization problem that steers the relative importance of multiple objectives through controllable weights during decoding, without modifying model parameters or requiring external reward models. Specifically, a reward discovery module constructs prompt-derived surrogate rewards from the backbone LLM for diverse objectives specified in natural language, while a weight optimization module dynamically adjusts objective weights based on the user's initial preferences and the partially generated response to balance competing objectives during generation. The resulting surrogate rewards and weights jointly guide an online optimization procedure over the token distribution, enabling better alignment with the target objectives. Extensive experiments across multiple datasets and backbone LLMs show that MATO consistently outperforms strong baselines, achieving Pareto-improving multi-objective alignment and stronger steerability. These results highlight test-time optimization as a promising direction for scalable, controllable, and model-agnostic personalized alignment.
A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
University of California, San Diego(加利福尼亚大学圣地亚哥分校);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Adobe Research(奥多比研究院);The Ohio State University(俄亥俄州立大学);Allen Institute for AI(艾伦人工智能研究所)
A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications
Ant Group(蚂蚁集团);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院)
From Generic Empathy to Personalized Emotional Support: A Self-Evolution Framework for User Preference Alignment
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS(中国科学院自动化研究所多模态人工智能系统国家重点实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment
The Hong Kong Polytechnic University(香港理工大学);Huawei Technologies Ltd.(华为技术有限公司);National University of Singapore(新加坡国立大学);University of Science and Technology of China(中国科学技术大学)
FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment
College of Intelligence and Computing, Tianjin University, Tianjin, China(智能与计算学院,天津大学,天津,中国);College of Computing and Data Science, Nanyang Technological University, Singapore(计算与数据科学学院,南洋理工大学,新加坡)
MAPS: Motivation-Aware Personalized Search via LLM-Driven Consultation Alignment
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);University of International Business and Economics(国际商务经济大学);AI Lab at Lenovo Research, Lenovo Group Limited(联想集团研究院人工智能实验室)
Multi-Objective Alignment of Language Models for Personalized Psychotherapy
Department of Computer Science, University of California, Los Angeles, CA, USA(计算机科学系,加州大学洛杉矶分校);Department of Psychology, The University of Texas at Austin, Austin, TX, USA(心理学系,德克萨斯大学奥斯汀分校);Department of Psychiatry, NYU Grossman School of Medicine, New York, NY, USA(精神病学系,纽约大学格罗斯曼医学院)
CASTLE: A Comprehensive Benchmark for Evaluating Student-Tailored Personalized Safety in Large Language Models
East China Normal University(华东师范大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Zhejiang University(浙江大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Education University of Hong Kong(香港教育大学)
BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization
Graduate School of AI, KAIST(韩国科学技术院人工智能研究生院);Samsung Advanced Institute of Technology(三星先进技术研究院);Department of Electrical and Computer Engineering, UBC(不列颠哥伦比亚大学电气与计算机工程系)
"In Dialogues We Learn": Towards Personalized Dialogue Without Pre-defined Profiles through In-Dialogue Learning
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Ant Group(蚂蚁集团);University of Electronic Science and Technology of China(电子科技大学);Kunming University of Science and Technology(昆明理工大学)
EduAgentQG: Multi-Agent Personalized Mathematics Question Generation with Explicit Diversity and Objective-Aware Evaluation
East China Normal University(东华大学);Shanghai Institute of Al for Education(上海人工智能教育研究院);The Chinese University of Hong Kong, Shenzhen School of Data Science(香港中文大学(深圳)数据科学学院)
Large Language Models Empowered Personalized Web Agents
National University of Singapore(新加坡国立大学);The Hong Kong Polytechnic University(香港理工大学);University of Science and Technology of China(中国科学技术大学);Eastern Institute of Technology(宁波东方理工大学)
From Critique to Clarity: A Pathway to Faithful and Personalized Code Explanations with Large Language Models
Institute for Clarity in Documentation(文献清晰度研究所);Inria Paris-Rocquencourt(法国国家信息与自动化研究所巴黎-罗康库尔中心);Rajiv Gandhi University(拉吉夫·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕尔默研究实验室);The Thørväld Group(索尔沃尔德集团);The Kumquat Consortium(金桔联盟)
DiverValue-Bench: A Benchmark and Fine-Tuning Framework for Aligning Large Language Models with Diverse Human Values
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Renmin University of China(中国人民大学);Beijing Institute of AI Safety and Governance(北京人工智能安全与治理研究院);University of Chinese Academy of Sciences(中国科学院大学)