arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.03467cs.CV

ThinkRL-Edit: 基于强化学习的图像编辑推理框架

ThinkRL-Edit: Thinking in Reinforcement Learning for Reasoning-Centric Image Editing

Zhejiang University(浙江大学)

Hengjia Li, Liming Jiang, Qing Yan, Yizhi Song, Hao Kang, Zichuan Liu, Xin Lu, Boxi Wu, Deng Cai

AI总结:

ThinkRL-Edit通过引入链式推理和无偏奖励策略,提升图像编辑的推理能力,实现更精准和稳定的编辑效果。

摘要:

基于统一多模态生成模型的指令驱动图像编辑技术取得了快速发展,但其底层视觉推理能力仍然有限,导致在以推理为中心的编辑任务中表现不佳。强化学习(RL)已被研究用于提升图像编辑的质量,但面临三个关键挑战:(1)受限于去噪随机性内的推理探索能力;(2)奖励融合偏差;(3)基于VLM的指令奖励不稳定。在本工作中,我们提出了ThinkRL-Edit,一个以推理为中心的RL框架,将视觉推理与图像合成解耦,并扩展推理探索范围超越去噪。为此,我们引入了基于链式推理(CoT)的推理采样方法,在在线采样前引入规划和反思阶段,迫使模型探索多个语义假设并验证其合理性后再确定视觉结果。为避免加权聚合的失败,我们提出了一种跨多个奖励维度的无偏链式偏好分组策略。此外,我们用二元检查清单替代基于区间值的VLM评分,获得更精确、方差更低且可解释的奖励用于复杂推理。实验表明,我们的方法在以推理为中心的图像编辑任务中显著优于先前工作,生成符合指令、视觉一致且语义稳固的编辑结果。

英文摘要:

Instruction-driven image editing with unified multimodal generative models has advanced rapidly, yet their underlying visual reasoning remains limited, leading to suboptimal performance on reasoning-centric edits. Reinforcement learning (RL) has been investigated for improving the quality of image editing, but it faces three key challenges: (1) limited reasoning exploration confined to denoising stochasticity, (2) biased reward fusion, and (3) unstable VLM-based instruction rewards. In this work, we propose ThinkRL-Edit, a reasoning-centric RL framework that decouples visual reasoning from image synthesis and expands reasoning exploration beyond denoising. To the end, we introduce Chain-of-Thought (CoT)-based reasoning sampling with planning and reflection stages prior to generation in online sampling, compelling the model to explore multiple semantic hypotheses and validate their plausibility before committing to a visual outcome. To avoid the failures of weighted aggregation, we propose an unbiased chain preference grouping strategy across multiple reward dimensions. Moreover, we replace interval-based VLM scores with a binary checklist, yielding more precise, lower-variance, and interpretable rewards for complex reasoning. Experiments show our method significantly outperforms prior work on reasoning-centric image editing, producing instruction-faithful, visually coherent, and semantically grounded edits.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑