BOTS：一种用于LLM强化微调中贝叶斯在线任务选择的统一框架
BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning
AI总结：
BOTS通过贝叶斯推断和汤普森采样，提高LLM强化微调中任务选择的效率和性能。
摘要：
强化微调（RFT）是将大语言模型（LLM）与人类偏好对齐并增强推理的关键技术，但其效果高度依赖于训练过程中探索的任务。均匀任务采样效率低下，浪费在琐碎或无法解决的任务上，而现有任务选择方法常面临高展开成本、适应性差或证据不完整的问题。我们引入BOTS，一种用于LLM强化微调中的贝叶斯在线任务选择的统一框架。基于贝叶斯推断，BOTS能够随着模型进化自适应地维护任务难度的后验估计。它联合纳入来自所选任务直接评估的显式证据以及对未选任务推断出的隐式证据，通过汤普森采样确保在任务选择中平衡探索与利用。为了使隐式证据实用化，我们将其实例化为一个超轻量的插件，该插件通过插值方法估计任务难度，无需额外展开，添加了可忽略的开销。实证结果表明，在多样化的领域和LLM规模下，BOTS在数据效率和性能上均优于基线和消融实验，提供了一种实用且可扩展的动态任务选择解决方案。代码可在https://github.com/agentscope-ai/Trinity-RFT/tree/main/examples/bots获取。
英文摘要：
Reinforcement finetuning (RFT) is a key technique for aligning Large Language Models (LLMs) with human preferences and enhancing reasoning, yet its effectiveness is highly sensitive to which tasks are explored during training. Uniform task sampling is inefficient, wasting computation on tasks that are either trivial or unsolvable, while existing task selection methods often suffer from high rollout costs, poor adaptivity, or incomplete evidence. We introduce BOTS, a unified framework for Bayesian Online Task Selection in LLM reinforcement finetuning. Grounded in Bayesian inference, BOTS adaptively maintains posterior estimates of task difficulty as the model evolves. It jointly incorporates explicit evidence from direct evaluations of selected tasks and implicit evidence inferred from these evaluations for unselected tasks, with Thompson sampling ensuring a principled balance between exploration and exploitation for task selection. To make implicit evidence practical, we instantiate it with an ultra-light interpolation-based plug-in that estimates difficulties of tasks without extra rollouts, adding negligible overhead. Empirically, across diverse domains and LLM scales, BOTS consistently improves data efficiency and performance over baselines and ablations, providing a practical and extensible solution for dynamic task selection in RFT. Code is available at https://github.com/agentscope-ai/Trinity-RFT/tree/main/examples/bots.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
