arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.20759cs.CL
收藏

奥顿多元强化学习用于大语言模型

Overton Pluralistic Reinforcement Learning for Large Language Models

University College London(伦敦大学学院);University of Basel(巴塞尔大学)

Yu Fu, Seongho Son, Ilija Bogunovic

AI总结:

本文提出OP-GRPO框架,通过强化学习使大语言模型生成多样化响应,提升人类视角覆盖和视角独特性,实验证明其在自然语言推理任务中的优越性能。

摘要:

现有对齐范式在捕捉人类价值观的多元性方面仍然有限。奥顿多元主义通过从单个查询生成多样化视角来解决这一缺口。本文介绍了OP-GRPO(奥顿多元主义群体相对策略优化),一种用于隐式奥顿多元主义的强化学习框架,使单个大语言模型能够生成多元化的响应,而无需显式提示或模块化编排。我们的工作流程包括两个主要步骤。首先,相似性估计器训练对Sentence Transformer进行微调,以适应奥顿多元主义任务,从而提供更准确的生成响应覆盖评估。其次,OP-GRPO训练将此相似性估计器纳入双奖励系统中,以确保真实人类视角的广泛覆盖和每个视角的独特性,从而促进多样性。实证结果展示了

英文摘要:

Existing alignment paradigms remain limited in capturing the pluralistic nature of human values. Overton Pluralism (OP) addresses this gap by generating responses with diverse perspectives from a single query. This paper introduces OP-GRPO (Overton Pluralistic Group Relative Policy Optimization), a reinforcement learning framework for implicit Overton Pluralism that enables a single LLM to produce pluralistic responses without explicit prompting or modular orchestration. Our workflow consists of two main steps: 1) Similarity estimator training, which fine-tunes a Sentence Transformer for OP tasks to provide a more accurate coverage evaluation of the given responses; and 2) OP-GRPO training, which incorporates this similarity estimator into a carefully designed dual-reward system to ensure both broad coverage of genuine human perspectives and the uniqueness of each perspective, thereby promoting diversity. Empirical results demonstrate that OP-GRPO achieves a "Small models, Big perspective coverage" effect: our trained Qwen2.5-3B-Instruct surpasses the GPT-OSS (20B) baseline with a 37.4% relative accuracy gain on the Natural Language Inference (NLI) benchmark. It also outperforms a modular-architecture baseline with a 19.1% relative improvement. Evaluations with GPT-5.4 as the LLM judge and OP-Benchmark further confirm the robustness of our approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑