In this paper, we investigate the impact of diverse user preference on learning under the stochastic multi-armed bandit (MAB) framework. We aim to show that when the user preferences are sufficiently diverse and each arm can be optimal for certain users, the O(log T) regret incurred by exploring the sub-optimal arms under the standard stochastic MAB setting can be reduced to a constant. Our intuition is that to achieve sub-linear regret, the number of times an optimal arm being pulled should scale linearly in time; when all arms are optimal for certain users and pulled frequently, the estimated arm statistics can quickly converge to their true values, thus reducing the need of exploration dramatically. We cast the problem into a stochastic linear bandits model, where both the users preferences and the state of arms are modeled as {independent and identical distributed (i.i.d)} d-dimensional random vectors. After receiving the user preference vector at the beginning of each time slot, the learner pulls an arm and receives a reward as the linear product of the preference vector and the arm state vector. We also assume that the state of the pulled arm is revealed to the learner once its pulled. We propose a Weighted Upper Confidence Bound (W-UCB) algorithm and show that it can achieve a constant regret when the user preferences are sufficiently diverse. The performance of W-UCB under general setups is also completely characterized and validated with synthetic data.
Versatile Dueling Bandits: Best-of-both-World Analyses for Online Learning from Preferences
Microsoft Research, NYC(微软研究院,纽约市);Univ. Grenoble Alpes(格勒诺布尔阿尔卑斯大学);Inria(法国国家信息与自动化研究所);CNRS(法国国家科学研究中心);Grenoble INP(格勒诺布尔综合理工学院)
我们研究的问题$K$武装决斗土匪随机和对抗环境中,学习者的目标是通过相对偏好的决策点查询在网上顺序的方式来聚合信息。我们首先提出了一个新的减少从任何(一般)决斗土匪多武装土匪,尽管简单,它使我们能够改善许多现有的结果决斗土匪。特别是,\emph{we give the first best-of-both world result for the dueling bandits regret minimization problem} --一个统一的框架,保证同时对随机和对抗性偏好进行最佳执行。此外,我们的算法也是第一个实现最佳的$O(\sum_{i = 1}^K \frac{\log T}{Δ_i})$遗憾约束对康多塞冠军的基准,最佳规模的手臂大小$K$和实例特定的次优差距$\{Δ_i\}_{i = 1}^K$。这解决了长期存在的问题,设计一个实例间隙相关的顺序最优后悔算法决斗土匪(匹配下限小常数因子)。我们进一步证明了我们提出的算法的鲁棒性,通过证明其最佳后悔率下adversarially损坏的偏好-这优于现有的最先进的损坏决斗结果的大幅度。总之,我们相信我们的减少想法将在解决不同类别的决斗土匪设置中找到更广泛的范围,否则将与通常更复杂的解决方案和更差的保证的多武装土匪分开研究。我们提出的算法的有效性是经验证实对现有的决斗土匪的方法。
Contextual Online Uncertainty-Aware Preference Learning for Human Feedback
Department of Biostatistics, Harvard T.H. Chan School of Public Health(哈佛大学T.H.陈公共卫生学院生物统计学系);Department of Biostatistics & Bioinformatics, Duke University(杜克大学生物统计学与生物信息学系)
T-POP: Test-Time Personalization with Online Preference Feedback
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));East China Normal University(华东师范大学);Shenzhen Loop Area Institute(深圳河套学院);Tianjin University(天津大学);The Chinese University of Hong Kong(香港中文大学);Beihang University(北京航空航天大学);City University of Hong Kong(香港城市大学);The Hong Kong University of Science and Technology (Guangzhou)(香港理工大学(广州))
National University of Defense Technology(国防科技大学);State Key Laboratory of Complex & Critical Software Environment(复杂与关键软件环境国家重点实验室);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Peng Cheng Laboratory(鹏城实验室)
Avoiding $\mathbf{exp(R_{max})}$ scaling in RLHF through Preference-based Exploration
Department of Electrical & Computer Engineering(电气与计算机工程系);Boston University(波士顿大学);Department of Computer Science(计算机科学系);Cornell University(康奈尔大学);Faculty of Computing & Data Sciences(计算与数据科学学院)
How Much Online RL is Enough? Informative Rollouts for Offline Preference Optimization in RLVR
TCS Research Department of CSE(TCS计算机科学系研究部);IIT Madras(印度理工学院马德拉斯分校);Department of Data Science & AI(数据科学与人工智能系);Wadhwani School of Data Science & AI(Wadhwani数据科学与人工智能学院)
Enhancing Bandit Algorithms with LLMs for Time-varying User Preferences in Streaming Recommendations
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学耿丽学校);School of Artificial Intelligence and Data Science, University of International Business and Economics(国际商务经济大学人工智能与数据科学学院)
MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment
Department of Statistics, Rutgers University, New Brunswick, United States;Department of Computer Science, Rutgers University, New Brunswick, United States;College of Management of Technology, EPFL, Switzerland;Department of Computer Science, ETH Zurich, Switzerland
Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
Department of Statistics and Data Science, The Chinese University of Hong Kong(统计与数据科学系,香港中文大学);Department of Statistics, University of Wisconsin-Madison(统计系,威斯康星大学麦迪逊分校)
Data-dependent Exploration for Online Reinforcement Learning from Human Feedback
Center for Advanced Intelligence Project, RIKEN(日本理化学研究所先进智能研究中心);Graduate School of Frontier Sciences, The University of Tokyo(东京大学前沿科学研究生院);Northeastern University(东北大学);Zhejiang Gongshang University(浙江工商大学)
LEASE: Offline Preference-based Reinforcement Learning with High Sample Efficiency
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);CASIA-MUST Joint Laboratory of Intelligence Science and Technology, Institute of Systems Engineering, Macau University of Science and Technology(智能科学与技术CASIA-MUST联合实验室,澳门科学理工学院)