arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.11549cs.HC
收藏

UNIPO:统一的交互式可视化用于RL微调策略优化

UNIPO: Unified Interactive Visual Explanation for RL Fine-Tuning Policy Optimization

Georgia Tech(佐治亚理工学院)

Aeree Cho, Alexander D. Greenhalgh, Jonathan Bodea, Anthony Peng, Duen Horng Chau

AI总结:

UNIPO通过统一设计揭示RL微调算法的token级训练动态,提供高阶训练概述、步骤级提示响应检查器和算法对比视图,支持非专家教学和AI从业者算法选择。

摘要:

强化学习已成为微调大语言模型行为的主导技术,策略优化(PO)算法如GRPO、DAPO和Dr. GRPO相继出现,推动了最先进的推理和对齐性能。然而,这些算法在剪切、优势估计和奖励聚合方面的模块差异在不同论文中以不一致的符号引入,使比较困难且令非专家望而却步。我们提出了UNIPO,第一个交互式可视化工具,通过统一设计暴露RL微调算法的token级训练动态。UNIPO连接三个互补的视图:高阶训练概述、步骤级提示和响应检查器以及并列算法对比,使学习者能够观察个体设计决策如何通过训练传播。通过两个使用场景,我们展示了UNIPO如何支持非专家的课堂教学和AI从业者的算法选择。我们的工具是开源的,并在https://poloclub.github.io/unipo公开可用。

英文摘要:

Reinforcement learning has emerged as a dominant technique for fine-tuning the behavior of large language models, with policy optimization (PO) algorithms such as GRPO, DAPO, and Dr. GRPO emerging in rapid succession to advance state-of-the-art reasoning and alignment performance. However, the modular differences between these algorithms, including targeted improvements to clipping, advantage estimation, and reward aggregation, are introduced across separate papers with inconsistent notation, making them difficult to compare and intimidating to the non-expert community. We present UNIPO, to our knowledge the first interactive visualization tool that exposes the token-level training dynamics of RL fine-tuning algorithms through a unified design. UNIPO connects three complementary views, a high-level training overview, a step-level prompt and response inspector, and a side-by-side algorithm comparison, allowing learners to observe how individual design decisions propagate through training. Through two usage scenarios, we demonstrate how UNIPO supports both classroom instruction for non-experts and algorithm selection for AI practitioners. Our tool is open-source and publicly available at https://poloclub.github.io/unipo.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑