Preference-based reinforcement learning can learn effective reward functions from comparisons, but its scalability is constrained by the high cost of oracle feedback. Lightweight vision-language embedding (VLE) models provide a cheaper alternative, but their noisy outputs limit their effectiveness as standalone reward generators. To address this challenge, we propose ROVED, a hybrid framework that combines VLE-based supervision with targeted oracle feedback. Our method uses the VLE to generate segment-level preferences and defers to an oracle only for samples with high uncertainty, identified through a filtering mechanism. In addition, we introduce a parameter-efficient fine-tuning method that adapts the VLE with the obtained oracle feedback in order to improve the model over time in a synergistic fashion. This ensures the retention of the scalability of embeddings and the accuracy of oracles, while avoiding their inefficiencies. Across multiple robotic manipulation tasks, ROVED matches or surpasses prior preference-based methods while reducing oracle queries by up to 80%. Remarkably, the adapted VLE generalizes across tasks, yielding cumulative annotation savings of up to 90%, highlighting the practicality of combining scalable embeddings with precise oracle supervision for preference-based RL.
Efficient Meta Reinforcement Learning for Preference-based Fast Adaptation
University of California, Los Angeles（加州大学洛杉矶分校）；Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；Helixon Ltd.（Helixon有限公司）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）
Center for Intelligent and Networked System (CFINS)（智能与网络化系统中心）；Department of Automation（自动化系）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Beijing Key Laboratory of Embodied Intelligence Systems（北京具身智能系统重点实验室）；Tsinghua University（清华大学）
CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries
Beijing Key Laboratory of Embodied Intelligence Systems, Department of Automation, Tsinghua University, Beijing, China（北京智能体智能系统重点实验室，自动化系，清华大学，北京，中国）；The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China（复杂系统认知与决策智能重点实验室，自动化研究所，中国科学院，北京，中国）
Optimal Contextual Bandits with Knapsacks under Realizability via Regression Oracles
Department of Mathematics, HKUST（香港科技大学数学系）；Department of Industrial Engineering and Decision Analytics, HKUST（香港科技大学工业工程与决策分析系）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深港协同创新研究院）
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Salesforce AI Research（Salesforce AI研究院）；The Hong Kong University of Science and Technology（香港科技大学）
Selective Preference Optimization via Token-Level Reward Function Estimation
National Centre for Text Mining, The University of Manchester（曼彻斯特大学文本挖掘中心）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；Center for Language and Information Research, Wuhan University（武汉大学语言与信息研究中心）；The Fin AI（Fin AI公司）；Baidu Inc.（百度公司）；Archimedes Research（阿基米德研究）
Truthful AI Advisors: A Pre-Specified Benchmark for Large Language Model Honesty Under Preference Misalignment
Amazon Lab126, HW Tech Org.（亚马逊实验室126，硬件技术组织）；Computational Modeling and Simulation University of Pittsburgh（计算建模与仿真大学匹兹堡分校）；Mathematics & Statistics Department University of Minnesota Duluth（数学与统计学系明尼苏达大学 Duluth 分校）
A Statistical Framework for Alignment with Biased AI Feedback
Center for Data Science, Zhejiang University（浙江大学数据科学中心）；Department of Electrical and Computer Engineering, Rutgers University（罗格斯大学电气与计算机工程系）；Department of Statistics, Rutgers University（罗格斯大学统计学系）；Faculty of Business and Economics, The University of Hong Kong（香港大学商学院）
