One Model for All: Multi-Objective Controllable Language Models
Ruhr University Bochum（波鸿鲁尔大学）；Eindhoven University of Technology（埃因霍温理工大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Liverpool（利物浦大学）
Aligning large language models (LLMs) with human preferences is critical for enhancing LLMs' safety, helpfulness, humor, faithfulness, etc. Current reinforcement learning from human feedback (RLHF) mainly focuses on a fixed reward learned from average human ratings, which may weaken the adaptability and controllability of varying preferences. However, creating personalized LLMs requires aligning LLMs with individual human preferences, which is non-trivial due to the scarce data per user and the diversity of user preferences in multi-objective trade-offs, varying from emphasizing empathy in certain contexts to demanding efficiency and precision in others. Can we train one LLM to produce personalized outputs across different user preferences on the Pareto front? In this paper, we introduce Multi-Objective Control (MOC), which trains a single LLM to directly generate responses in the preference-defined regions of the Pareto front. Our approach introduces multi-objective optimization (MOO) principles into RLHF to train an LLM as a preference-conditioned policy network. We improve the computational efficiency of MOC by applying MOO at the policy level, enabling us to fine-tune a 7B-parameter model on a single A6000 GPU. Extensive experiments demonstrate the advantages of MOC over baselines in three aspects: (i) controllability of LLM outputs w.r.t. user preferences on the trade-off among multiple rewards; (ii) quality and diversity of LLM outputs, measured by the hyper-volume of multiple solutions achieved; and (iii) generalization to unseen preferences. These results highlight MOC's potential for real-world applications requiring scalable and customizable LLMs.
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Tencent Inc.（腾讯公司）
Large Language Models for Controllable Multi-property Multi-objective Molecule Optimization
Department of Computer Science and Engineering, The Ohio State University, USA（计算机科学与工程系，俄亥俄州立大学）；Translational Data Analytics Institute, The Ohio State University, USA（转化数据分析研究所，俄亥俄州立大学）；Department of Biomedical Informatics, The Ohio State University, USA（生物医学信息学系，俄亥俄州立大学）；College of Pharmacy, The Ohio State University, USA（药学院，俄亥俄州立大学）
Controllable Generation from Pre-trained Language Models via Inverse Prompting
Tsinghua University（清华大学）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）；Recurrent AI, Ltd.（Recurrent AI 有限公司）；DAMO Academy, Alibaba Inc.（阿里巴巴达摩院）
Unveiling the Potential of Diffusion Large Language Model in Controllable Generation
University of Southern California（南加州大学）；University of Queensland（昆士兰大学）；University of California, San Diego（加州大学圣地亚哥分校）；University of California, Merced（加州大学默塞德分校）
DiffLM: Controllable Synthetic Data Generation via Diffusion Language Models
University of Chinese Academy of Sciences（中国科学院大学）；Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所信息处理实验室）；ByteDance Inc.（字节跳动公司）
Quark: Controllable Text Generation with Reinforced Unlearning
Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Paul G. Allen School of Computer Science, University of Washington（华盛顿大学保罗·G·艾伦计算机科学学院）
大规模的语言模型经常学习与用户期望不一致的行为。生成的文本可能包含攻击性或有毒语言，包含大量重复，或者与用户期望的不同情绪。我们考虑的任务是通过微调语言模型的信号不做什么来消除这些错位。我们引入了量化奖励约束（Quark），这是一种优化奖励函数的算法，该函数量化了（不）想要的属性，同时不会偏离原始模型太远。Quark在（i）收集具有当前语言模型的样本，（ii）基于奖励将它们分类为分位数，每个分位数由预先添加到语言模型的输入的奖励令牌标识，以及（iii）在来自每个分位数的以其奖励令牌为条件的样本上使用标准语言建模损失，同时通过KL发散惩罚保持在原始语言模型附近。通过在生成时以高奖励令牌为条件，该模型生成的文本显示出较少的不需要的属性。对于消除毒性，消极情绪和重复，我们的实验表明Quark优于强基线和最先进的强化学习方法，如PPO（Schulman et al. 2017），同时仅依赖于标准语言建模原语。
From Rational Answers to Emotional Resonance: The Role of Controllable Emotion Generation in Language Models
State Key Laboratory on Technologies for Chinese Medicine Pharmaceutical Process Control and Intelligent Manufacture（中药制药过程控制与智能制造技术国家重点实验室）；Zhejiang Lab（浙江实验室）
Reasoning Up the Instruction Ladder for Controllable Language Models
Department of Computer Science and Engineering, The Ohio State University（俄亥俄州立大学计算机科学与工程系）；Microsoft Research（微软研究院）；Allen Institute for AI（艾伦人工智能研究所）
From Words to Widgets for Controllable LLM Generation
Cornell University（康奈尔大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Loyola University Maryland（路易斯安那大学马里兰分校）
Scaffold-Conditioned Preference Triplets for Controllable Molecular Optimization with Large Language Models
National Key Laboratory of Parallel and Distributed Processing（平行与分布式处理国家重点实验室）；College of Computer Science and Technology（计算机科学与技术学院）；National University of DefenseTechnology（国防科技大学）；DP Technology（DP科技）
Evolutionary Soups: Evolving Mixture-of-Experts for Multi-Objective LLM Alignment
Fraunhofer Institute for Applied Information Technology FIT（弗劳恩霍夫应用信息技术研究所FIT）；University of Cologne（科隆大学）；University of Stuttgart（斯图加特大学）；University of Southampton（南安普顿大学）；Soochow University（苏州大学）；University Hospital of Cologne（科隆大学医院）
Exploring Multilingual Concepts of Human Value in Large Language Models: Is Value Alignment Consistent, Transferable and Controllable across Languages?
School of New Media and Communication, Tianjin University（天津大学新媒体与传播学院）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）
Goal-conditioned GFlowNets for Controllable Multi-Objective Molecular Design
Mila（米拉研究所）；École Polytechnique de Montréal（蒙特利尔综合理工学院）；Université de Montréal（蒙特利尔大学）；ServiceNow（ServiceNow公司）；Recursion Pharmaceuticals（Recursion制药公司）
