HiPO: Hierarchical Preference Optimization for Adaptive Reasoning in LLMs
Vellore Institute of Technology（维洛雷理工学院）；University of Massachusetts Amherst（马萨诸塞大学阿姆赫斯特分校）；Northwestern University（西北大学）；Yale University（耶鲁大学）；Algoverse AI Research（Algoverse AI研究）
12 pages, 4 figures, 6 tables. Includes ablation study across Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct on 5 math reasoning benchmarks (GSM8K, MATH500, Minerva, AIME24, Gaokao2023). GPT-4.1 used for structured evaluation of reasoning quality
Direct Preference Optimization (DPO) is an effective framework for aligning large language models with human preferences, but it struggles with complex reasoning tasks. DPO optimizes for the likelihood of generating preferred over dispreferred responses in their entirety and lacks the granularity to provide feedback on subsections of many-step solutions typical of reasoning tasks. Existing methods excel at either stable preference learning (e.g., DPO variants like KTO and RSO) or structured reasoning (e.g., ReMA's multi-agent RL framework, Tree of Thoughts), but fail to merge these complementary strengths. We propose HiPO (Hierarchical Preference Optimization), an extension of DPO that separates responses into reasoning segments (query clarification and context, reasoning steps, and answer) and computes loss as a weighted sum of the DPO loss for each segment. Our approach enables segment-specific training while maintaining DPO's computational efficiency and training stability. We demonstrate that for multiple 7B LLMs fine-tuned using HiPO and DPO on the Math Stack Exchange preference dataset, the models trained with HiPO outperform the others on a variety of common math benchmarks and achieve greater organization, logical flow, and consistency as measured by GPT-4.1.
HIPO: Instruction Hierarchy via Constrained Reinforcement Learning
School of ECEE Arizona State University（亚利桑那州立大学电子与计算机工程学院）；Department of ECE The Ohio State University（俄亥俄州立大学电子工程系）；Computer Science Department University of Houston（休斯顿大学计算机科学系）；College of Engineering & Applied Science CU Boulder（科罗拉多大学博尔德分校工程与应用科学学院）；Dept. of Electrical Engineering & Computer Science United States Military Academy（美国军事学院电子工程与计算机科学系）
Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（人工智能与数字经济广东实验室）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）；Baidu Inc.（百度公司）
Fairness Begins with State: Purifying Latent Preferences for Hierarchical Reinforcement Learning in Interactive Recommendation
Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences（中国科学院重庆绿色智能技术研究所）；Chongqing School, University of Chinese Academy of Sciences（中国科学院大学重庆学院）；The First Affiliated Hospital, University of Science and Technology of China（中国科学技术大学第一附属医院）；City University of Hong Kong（香港城市大学）
Patient-Centered Treatment Planning for Chronic Multimorbidity: A Hierarchical Reinforcement Learning Framework for Preference Modeling
University of Tennessee, Knoxville（田纳西大学诺克斯维尔分校）；University of Tennessee Health Science Center College of Medicine（田纳西大学健康科学中心医学院）；University of Tennessee Medical Center（田纳西大学医学中心）；University of Arkansas（阿肯色大学）
Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach
IIT Kanpur（印度理工学院坎浦尔分校）；University of Maryland, College Park（马里兰大学学院公园分校）；U.S. Army Research Laboratory（美国陆军研究实验室）；University of Texas, Austin（德克萨斯大学奥斯汀分校）；Oracle（Oracle公司）；University of Central Florida（中央佛罗里达大学）；University of Bath（巴斯大学）
Multi-objective Large Language Model Alignment with Hierarchical Experts
Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学（深圳））；Nanyang Technological University（南洋理工大学）；Peng Cheng Laboratory（鹏城实验室）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Tencent（腾讯）
SPLASH! Sample-efficient Preference-based inverse reinforcement learning for Long-horizon Adversarial tasks from Suboptimal Hierarchical demonstrations
Boston University（波士顿大学）；MIT Lincoln Laboratory（麻省理工学院林肯实验室）；MIT Pavlab（麻省理工学院 Pavlab 实验室）；Woods Hole Oceanographic Institution（伍兹霍尔海洋研究所）；University of Maryland, College Park（马里兰大学学院公园分校）
ALaRM: Align Language Models via Hierarchical Rewards Modeling
Institute of Science and Technology for Brain-inspired Intelligence, Fudan University（复旦大学脑智能科学与技术研究院）；School of Data Science, Fudan University（复旦大学数据科学学院）；School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Research Institute of Intelligent and Complex Systems, Fudan University（复旦大学智能与复杂系统研究院）
Decoupling Strategy and Execution in Task-Focused Dialogue via Goal-Oriented Preference Optimization
School of Information Engineering, China Jiliang University, Hangzhou, China（信息工程学院，中国浙江大学，杭州，中国）；College of Chemical and Biological Engineering, Zhejiang University, Hangzhou, China（化学与生物工程学院，浙江大学，杭州，中国）
S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHF
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；School of Computer Science and Artificial Intelligence, Zhengzhou University（郑州大学计算机科学与人工智能学院）
