Large language model reasoning is often treated as a monolithic capability, relying on binary preference supervision that fails to capture partial progress or fine-grained reasoning quality. We introduce continuous utility direct preference optimization (CU-DPO), a framework that aligns models to a portfolio of prompt-based cognitive strategies by replacing binary labels with continuous scores that capture fine-grained reasoning quality. We prove that learning with K strategies yields a Theta(K log K) improvement in sample complexity over binary preferences and that DPO converges to the entropy-regularized utility-maximizing policy. To exploit this signal, we propose a two-stage pipeline: (i) strategy selection, which optimizes the model to choose the best strategy via best-vs-all comparisons, and (ii) execution refinement, which trains correct execution using margin-stratified pairs. The framework is domain-agnostic: any task admitting cognitively distinct solution strategies and a decomposable continuous utility signal can be incorporated into the portfolio. On mathematical reasoning benchmarks, CU-DPO improves strategy selection accuracy from 35-46% to 68-78% across seven base models, yielding downstream reasoning gains of up to +6.6 points on in-distribution datasets with effective out-of-distribution transfer. CU-DPO demonstrates consistent gains on code generation and causal reasoning benchmarks, confirming generalization beyond the mathematical domain.
Preference Elicitation for Policy Optimization and Application to Aligning Heart Transplantation with Human Values
Carnegie Mellon University(卡内基梅隆大学);Medical University of South Carolina(南卡罗来纳医科大学);Strategy Robot, Inc.(策略机器人公司);Strategic Machine, Inc.(战略机器公司);Optimized Markets, Inc.(优化市场公司)
Adaptive Preference Optimization with Uncertainty-aware Utility Anchor
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China(认知智能国家重点实验室,中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(人工智能研究院,合肥综合性国家科学中心);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,BIGAI)
Learning Economic Parameters from Revealed Preferences
Carnegie Mellon University(卡内基梅隆大学);The Hebrew University(希伯来大学);Georgia Institute of Technology(佐治亚理工学院)
从Beigman and Vohra(2006)和Zadimoghoughs and Roth(2012)开始,最近的一系列工作解决了{ \em学习}从显示的偏好数据中获得效用函数的问题。这里的目标是利用过去的数据描述购买的效用最大化代理时,面对一定的价格和预算约束,以产生一个假设函数,可以准确地预测{ \em未来}的代理行为。在这项工作中,我们通过为一些重要的类提供样本复杂性保证和有效的算法来推进这一工作。通过与多类学习的最新进展相联系,我们提供了一种计算效率高的算法,该算法具有严格的样本复杂度保证($Θ(d/ε)$用于$d$商品的情况),用于在线性价格模型下学习线性效用函数。这就解决了Zadimoghloves and Roth(2012)中的一个悬而未决的问题。我们的技术产生了许多概括,包括学习其他研究良好的效用函数类的能力,以处理一个错误指定的模型,并与非线性价格。
Active Preference Learning over Latent Preference Archetypes for Many-Objective Bayesian Optimization
University of Manchester(曼彻斯特大学);University of Edinburgh(爱丁堡大学);Aalto University(阿尔托大学);University of Amsterdam(阿姆斯特丹大学);ELLIS Institute Finland(芬兰ELLIS研究所)
Learning Personalized Thermal Preferences via Bayesian Active Learning with Unimodality Constraints
Purdue University(普渡大学);School of Mechanical Engineering, Purdue University(普渡大学机械工程学院);Lyles School of Civil Engineering, Purdue University(普渡大学莱尔工程学院);Center for High Performance Buildings, Ray W. Herrick Laboratories, Purdue University(普渡大学雷·W·赫里克实验室高性能建筑中心)
From Social to Individuals: a Parsimonious Path of Multi-level Models for Crowdsourced Preference Aggregation
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Tencent AI Lab(腾讯人工智能实验室);Peking University(北京大学);University of Chinese Academy of Sciences(中国科学院大学);Hong Kong University of Science and Technology(香港科技大学)
Parsimonious Mixed-Effects HodgeRank for Crowdsourced Preference Aggregation
State Key Laboratory of Information Security, Institute of Information Engineering, Chinese Academy of Sciences(信息安全国家重点实验室,中国科学院信息工程研究所);Peking University(北京大学)