Aligning language models (LMs) to human preferences has emerged as a critical pursuit, enabling these models to better serve diverse user needs. Existing methods primarily focus on optimizing LMs for a single reward function, limiting their adaptability to varied objectives. Here, we propose $\textbf{multi-objective decoding (MOD)}$, a decoding-time algorithm that outputs the next token from a linear combination of predictions of all base models, for any given weightings over different objectives. We exploit a common form among a family of $f$-divergence regularized alignment approaches (such as PPO, DPO, and their variants) to identify a closed-form solution by Legendre transform, and derive an efficient decoding strategy. Theoretically, we show why existing approaches can be sub-optimal even in natural settings and obtain optimality guarantees for our method. Empirical results demonstrate the effectiveness of the algorithm. For example, compared to a parameter-merging baseline, MOD achieves 12.8% overall reward improvement when equally optimizing towards $3$ objectives. Moreover, we experiment with MOD on combining three fully-finetuned LLMs of different model sizes, each aimed at different objectives such as safety, coding, and general user preference. Unlike traditional methods that require careful curation of a mixture of datasets to achieve comprehensive improvement, we can quickly experiment with preference weightings using MOD to find the best combination of models. Our best combination reduces toxicity on Toxigen to nearly 0% and achieves 7.9--33.3% improvement across other three metrics ($\textit{i.e.}$, Codex@1, GSM-COT, BBH-COT).
Fast Whole-Brain, Geometry-Aware Functional Alignment for Cross-Subject Decoding
Université Paris-Saclay(巴黎-萨克雷大学);Inria(法国国家信息与自动化技术研究院);CEA(法国原子能委员会);Cognitive Neuroimaging Unit(认知神经成像单位);Institut National de la Santé et de la Recherche Médicale(法国国家卫生与医学研究院);Commissariat à l’Energie Atomique et aux énergies alternatives(法国原子能委员会及替代能源委员会);NeuroSpin center(神经科学中心);Institut de neuromodulation(神经调制研究所);Université Paris Cité(巴黎-城市大学)
Cross-Session Decoding of Neural Spiking Data via Task-Conditioned Latent Alignment
Institute of Automation, Chinese Academy of Sciences(自动化研究所,中国科学院);Departments of Ophthalmology and Bioengineering, University of Pittsburgh(眼科学与生物工程系,匹兹堡大学);Inria, CEA, Université Paris-Saclay(法国国家科学研究中心(Inria)、法国国家科学研究中心(CEA)、巴黎萨克雷大学)
AFPM: Alignment-based Frame Patch Modeling for Cross-Dataset EEG Decoding
Key Laboratory of the Ministry of Education for Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology(教育部图像处理与智能控制重点实验室,人工智能与自动化学院,华中科技大学);Zhongguancun Academy(中关村学院)
Robust Multi-Objective Controlled Decoding of Large Language Models
University College London(伦敦大学学院);Ulsan National Institute of Science and Technology(釜山国立科学技术研究所);University of Basel(巴塞尔大学);University College London AI Centre(伦敦大学学院人工智能中心)
A Survey on Training-free Alignment of Large Language Models
School of Computer Science, Wuhan University, Wuhan, China(武汉大学计算机学院);Faculty of Computer Science and Technology, Qilu University of Technology, Shandong, China(青岛科技大学计算机科学与技术学院);Zhongguancun Academy, Beijing, China(中关村学院)
Stable Neural Decoding Across Sessions via Task-Conditioned Latent Alignment for Brain-Machine Interfaces
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Pittsburgh(匹兹堡大学);Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
fMRI2GES: Co-speech Gesture Reconstruction from fMRI Signal with Dual Brain Decoding Alignment
College of Information Science and Engineering, Hunan University(湖南大学信息科学与工程学院);School of Computer Science and Technology, Nanjing University of Science and Technology(南京理工大学计算机科学与技术学院);Department of Automation, Tsinghua University(清华大学自动化系)
REALM: Retrospective Encoder Alignment for LFP Modeling
Department of Biomedical Engineering, The Ohio State University(生物医学工程系,俄亥俄州立大学);Department of Information Technology, Monash University Malaysia(信息技术系,墨尔本大学马来西亚分校);NeuroTech Insititude, Columbus, OH, United States(神经科技研究所,美国哥伦布)
Multi-Objective Alignment of Language Models for Personalized Psychotherapy
Department of Computer Science, University of California, Los Angeles, CA, USA(计算机科学系,加州大学洛杉矶分校);Department of Psychology, The University of Texas at Austin, Austin, TX, USA(心理学系,德克萨斯大学奥斯汀分校);Department of Psychiatry, NYU Grossman School of Medicine, New York, NY, USA(精神病学系,纽约大学格罗斯曼医学院)