交互式批评-修订训练用于可靠的结构化大语言模型生成
Interactive Critique-Revision Training for Reliable Structured LLM Generation
AI总结：
本文提出DPA-GRPO方法，通过生成器-验证器游戏中的结构化验证干预，提升结构化决策的准确性与一致性，实验显示其在TaxCalcBench TY24上优于零样本生成和生成器-only RL基线。
摘要：
在结构化决策流程中，如表单填写、合规检查和维护报告，LLM输出必须本地正确、全局一致且可审计。现有优化方法常依赖启发式辩论、自我对战或LLM生成监督，导致二次保证问题。本文提出DPA-GRPO（双配对动作组相对策略优化），一种双玩家生成器-验证器游戏的配对动作训练方法，通过结构化验证干预。生成器提出输出并在受挑战时进行修订；验证器要么保持沉默或提出包含声明、论点和证据的安全保证案例（SAC）。SAC/no-SAC和KEEP/REVISE决定诱导配对反事实动作组，DPA-GRPO用于角色特定的KL正则化GRPO更新。分析无正则化游戏并证明严格低收益干预或修订动作的概率会产生单方面偏离。在标准随机逼近假设下，DPA-GRPO跟踪相应游戏ODE，其孤立渐近稳定的极限点是角色局部最优的稳态和候选局部均衡。在TaxCalcBench TY24实验中，DPA-GRPO在Qwen3-4B和Qwen3-8B上优于零样本生成和生成器-only RL基线，提升结构化决策准确性，增加正确沉默接受，减少遗漏错误，改善校准修订行为，表明生成器和验证器均获益。
英文摘要：
In structured decision-making workflows such as form filling, compliance checking, and maintenance reporting, LLM outputs must be locally correct, globally consistent, and auditable against task-specific rules. Existing refinement methods often rely on heuristic debate, self-play, or LLM-generated supervision, creating a second-order assurance problem. We propose DPA-GRPO (Dual Paired-Action Group-Relative Policy Optimization), a paired-action training method for a two-player generator--verifier game with structured verifier interventions. The generator proposes outputs and may revise them when challenged; the verifier either remains silent or raises a safety assurance case (SAC) containing a claim, argument, and evidence. These SAC/no-SAC and KEEP/REVISE decisions induce paired counterfactual action groups, which DPA-GRPO uses for role-specific KL-regularized GRPO updates. We analyze the unregularized game and show that positive probability on strictly lower-reward intervention or revision actions creates a profitable unilateral deviation. Under standard stochastic-approximation assumptions, DPA-GRPO tracks the corresponding game ODE, whose isolated asymptotically stable limit points are stationary and candidate local equilibria under role-wise local optimality. Experiments on TaxCalcBench TY24 show that DPA-GRPO improves structured decision accuracy over zero-shot generation and generator-only RL baselines across Qwen3-4B and Qwen3-8B. Training increases correct silent acceptance, reduces missed errors, and improves calibrated revision behavior, indicating gains for both generator and verifier.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
