图像描述的组相对策略优化
Group Relative Policy Optimization for Image Captioning
AI总结：
针对图像描述中SCST算法方差高、易陷入局部最优的问题，提出使用GRPO算法进行第二阶段优化，通过组内多候选比较和KL散度约束提升模型稳定性与生成多样性。
摘要：
图像描述任务通常使用两阶段训练来完成模型优化。第一阶段使用交叉熵作为损失函数进行优化，第二阶段使用自批判序列训练（SCST）进行强化学习优化。然而，SCST算法存在某些缺陷。SCST仅依赖单一的贪婪解码结果作为基线。如果模型本身不够稳定，贪婪解码结果可能相对最差，这将导致优势估计的方差较高，进而导致策略更新不稳定。此外，SCST仅将一个采样结果与贪婪解码结果进行比较，生成多样性有限，可能陷入局部最优。在本文中，我们提出使用最新的组相对策略优化（GRPO）强化学习算法作为第二阶段的优化方案。GRPO为输入图像生成多个候选描述，然后通过组内比较持续优化模型。通过约束策略更新的幅度和KL散度，大大保证了模型在训练期间的稳定性。此外，与仅采样一个答案的SCST相比，GRPO采样并生成多个答案。组内的多个候选答案覆盖了更广泛的解空间。结合KL散度约束，GRPO能在保证模型稳定性的同时提高多样性。本文代码可在https://github.com/liangxu-one/ms-models/tree/image_caption_grpo/research/arxiv_papers/Image_Caption_GRPO获取。
英文摘要：
Image captioning tasks usually use two-stage training to complete model optimization. The first stage uses cross-entropy as the loss function for optimization, and the second stage uses self-critical sequence training (SCST) for reinforcement learning optimization. However, the SCST algorithm has certain defects. SCST relies only on a single greedy decoding result as a baseline. If the model itself is not stable enough, the greedy decoding result may be relatively worst, which will lead to a high variance of advantage estimation, further leading to unstable policy updates. In addition, SCST only compares one sampling result with the greedy decoding result, and the generation diversity is limited, which may fall into a local optimum. In this paper, we propose using the latest Group Relative Policy Optimization (GRPO) reinforcement learning algorithm as an optimization solution for the second stage. GRPO generates multiple candidate captions for the input image and then continuously optimizes the model through intragroup comparison. By constraining the amplitude of policy updates and KL divergence, the stability of the model during training is greatly guaranteed. In addition, compared to SCST, which only samples one answer, GRPO samples and generates multiple answers. Multiple candidate answers in the group cover a wider solution space. Combined with KL divergence constraints, GRPO can improve diversity while ensuring model stability. The code for this article is available at https://github.com/liangxu-one/ms-models/tree/image_caption_grpo/research/arxiv_papers/Image_Caption_GRPO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
