Multi-Objective Exploration and Preference Optimization via Mutual Information
School of Computer, Beihang University(北京航空航天大学计算机学院);Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd(星辰AGI实验室,中国电信人工智能技术(北京)有限公司)
Aligning large language models with diverse and heterogeneous human values requires multi-objective alignment methods to effectively trade off conflicting preference dimensions. Current methods achieve this trade-off by training policies conditioned on preference vectors and leveraging online direct preference optimization. However, exploration uncertainty can cause the reward distributions of responses generated under different preference vectors to overlap, and the generated responses may fail to effectively align with the corresponding preference vectors. In this paper, we propose Multi-Objective Exploration and Preference Optimization via Mutual Information (MI-EPO), an information-theoretic framework. It unifies multi-objective exploration and alignment by maximizing the joint conditional mutual information among generated responses, preference feedback, and preference vectors. By incorporating a probabilistic routing mechanism, MI-EPO naturally decomposes objective alignment and preference-aware exploration, encouraging the model to generate responses that are distinguishable and aligned with different preference conditions. Experiments on safe alignment and helpful assistant tasks show that MI-EPO significantly improves the alignment between generated responses and preference vectors, makes the outputs more controllable, and achieves stable trade-offs across multiple objectives.
Mining or Synthesis? Rethinking Exploration Efficiency in Iterative Alignment of Mathematical Reasoning
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学深圳研究院);Huawei Large Model Data Technology Lab(华为大模型数据技术实验室);Huawei Multimodal Model Lab(华为多模态模型实验室);Department of Statistics and Data Science, Tsinghua University, Beijing, China(清华大学统计与数据科学系)
Mutual Enhancement of Large Language and Reinforcement Learning Models through Bi-Directional Feedback Mechanisms: A Planning Case Study
Technical University of Munich(慕尼黑工业大学);Peking University(北京大学);BIGAI(北京智源人工智能研究院);King's College London(伦敦国王学院);Tongji University(同济大学);University College London(伦敦大学学院)
Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
Department of Statistics and Data Science, The Chinese University of Hong Kong(统计与数据科学系,香港中文大学);Department of Statistics, University of Wisconsin-Madison(统计系,威斯康星大学麦迪逊分校)
Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
School of Computer Science and Engineering(计算机科学与工程学院);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications(新一代人工智能技术及其交叉应用关键实验室);Ministry of Education(教育部)
SceneWeave: Audience-Guided Scene Exploration for Dramatic Adaptation
The Central Academy of Drama(中央戏剧学院);Hithink RoyalFlush Information Network Co., Ltd.(同花顺信息技术股份有限公司);China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd.(中国电信人工智能技术(北京)有限公司);Beijing Institute of Mathematical Sciences and Applications (BIMSA)(北京数学科学与应用研究院)
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Tsinghua University(清华大学);Zhipu AI(智谱AI)