Reinforcement learning with human feedback~(RLHF) is critical for aligning Large Language Models (LLMs) with human preference. Compared to the widely studied offline version of RLHF, \emph{e.g.} direct preference optimization (DPO), recent works have shown that the online variants achieve even better alignment. However, online alignment requires on-the-fly generation of new training data, which is costly, hard to parallelize, and suffers from varying quality and utility. In this paper, we propose a more efficient data exploration strategy for online preference tuning (OPTune), which does not rely on human-curated or pre-collected teacher responses but dynamically samples informative responses for on-policy preference alignment. During data generation, OPTune only selects prompts whose (re)generated responses can potentially provide more informative and higher-quality training signals than the existing responses. In the training objective, OPTune reweights each generated response (pair) by its utility in improving the alignment so that learning can be focused on the most helpful samples. Throughout our evaluations, OPTune'd LLMs maintain the instruction-following benefits provided by standard preference tuning whilst enjoying 1.27-1.56x faster training speed due to the efficient data exploration strategy.
Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization
School of Computer Science and Engineering, Southeast University, Nanjing, China（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及其交叉应用国家重点实验室）
Align Generative Artificial Intelligence with Human Preferences: A Novel Large Language Model Fine-Tuning Method for Online Review Management
Department of Information Systems and Operations Management, The University of Texas at Arlington（信息系统与运营管理系，德克萨斯大学阿灵顿分校）；Department of Management Information Systems, University of Arizona（管理信息学系，亚利桑那大学）
T-POP: Test-Time Personalization with Online Preference Feedback
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；East China Normal University（华东师范大学）；Shenzhen Loop Area Institute（深圳河套学院）；Tianjin University（天津大学）；The Chinese University of Hong Kong（香港中文大学）；Beihang University（北京航空航天大学）；City University of Hong Kong（香港城市大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港理工大学（广州））
National University of Defense Technology（国防科技大学）；State Key Laboratory of Complex & Critical Software Environment（复杂与关键软件环境国家重点实验室）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Peng Cheng Laboratory（鹏城实验室）
Contextual Online Uncertainty-Aware Preference Learning for Human Feedback
Department of Biostatistics, Harvard T.H. Chan School of Public Health（哈佛大学T.H.陈公共卫生学院生物统计学系）；Department of Biostatistics & Bioinformatics, Duke University（杜克大学生物统计学与生物信息学系）
How Much Online RL is Enough? Informative Rollouts for Offline Preference Optimization in RLVR
TCS Research Department of CSE（TCS计算机科学系研究部）；IIT Madras（印度理工学院马德拉斯分校）；Department of Data Science & AI（数据科学与人工智能系）；Wadhwani School of Data Science & AI（Wadhwani数据科学与人工智能学院）
Enhancing Bandit Algorithms with LLMs for Time-varying User Preferences in Streaming Recommendations
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学耿丽学校）；School of Artificial Intelligence and Data Science, University of International Business and Economics（国际商务经济大学人工智能与数据科学学院）
Versatile Dueling Bandits: Best-of-both-World Analyses for Online Learning from Preferences
Microsoft Research, NYC（微软研究院，纽约市）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔综合理工学院）
我们研究的问题$K$武装决斗土匪随机和对抗环境中，学习者的目标是通过相对偏好的决策点查询在网上顺序的方式来聚合信息。我们首先提出了一个新的减少从任何（一般）决斗土匪多武装土匪，尽管简单，它使我们能够改善许多现有的结果决斗土匪。特别是，\emph{we give the first best-of-both world result for the dueling bandits regret minimization problem} --一个统一的框架，保证同时对随机和对抗性偏好进行最佳执行。此外，我们的算法也是第一个实现最佳的$O(\sum_{i = 1}^K \frac{\log T}{Δ_i})$遗憾约束对康多塞冠军的基准，最佳规模的手臂大小$K$和实例特定的次优差距$\{Δ_i\}_{i = 1}^K$。这解决了长期存在的问题，设计一个实例间隙相关的顺序最优后悔算法决斗土匪（匹配下限小常数因子）。我们进一步证明了我们提出的算法的鲁棒性，通过证明其最佳后悔率下adversarially损坏的偏好-这优于现有的最先进的损坏决斗结果的大幅度。总之，我们相信我们的减少想法将在解决不同类别的决斗土匪设置中找到更广泛的范围，否则将与通常更复杂的解决方案和更差的保证的多武装土匪分开研究。我们提出的算法的有效性是经验证实对现有的决斗土匪的方法。
Panacea: Pareto Alignment via Preference Adaptation for LLMs
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；BIGAI（北京智源人工智能研究院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；Yuanpei College, Peking University（北京大学元培学院）
Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
Department of Statistics and Data Science, The Chinese University of Hong Kong（统计与数据科学系，香港中文大学）；Department of Statistics, University of Wisconsin-Madison（统计系，威斯康星大学麦迪逊分校）
Data-dependent Exploration for Online Reinforcement Learning from Human Feedback
Center for Advanced Intelligence Project, RIKEN（日本理化学研究所先进智能研究中心）；Graduate School of Frontier Sciences, The University of Tokyo（东京大学前沿科学研究生院）；Northeastern University（东北大学）；Zhejiang Gongshang University（浙江工商大学）
Avoiding $\mathbf{exp(R_{max})}$ scaling in RLHF through Preference-based Exploration
Department of Electrical & Computer Engineering（电气与计算机工程系）；Boston University（波士顿大学）；Department of Computer Science（计算机科学系）；Cornell University（康奈尔大学）；Faculty of Computing & Data Sciences（计算与数据科学学院）
LEASE: Offline Preference-based Reinforcement Learning with High Sample Efficiency
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；CASIA-MUST Joint Laboratory of Intelligence Science and Technology, Institute of Systems Engineering, Macau University of Science and Technology（智能科学与技术CASIA-MUST联合实验室，澳门科学理工学院）
