AMR-SD：不对称元反射自蒸馏用于标记级信用分配
AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignment
AI总结：
本文提出AMR-SD，一种不对称元反射自蒸馏方法，旨在解决大型语言模型在复杂推理中因统一序列奖励导致的信用分配瓶颈问题，通过引入反思瓶颈和因果信息增益机制，实现更精确的标记级优势调节。
摘要：
大型语言模型（LLM）在复杂推理中的对齐严重依赖于可验证奖励强化学习（RLVR）。然而，标准算法如GRPO将序列级奖励均匀应用于所有标记，造成严重的信用分配瓶颈。尽管在线自我蒸馏试图通过将自我教师条件于特权上下文来解决这一问题，但直接暴露于原始 oracle 解决方案往往会诱导过条件的教师分布、隐含答案泄漏和晚期训练崩溃。为了克服这些限制，我们提出了不对称元反射自蒸馏（AMR-SD）。与直接条件于原始参考轨迹不同，AMR-SD插入了一个反思瓶颈：它将来自验证者结果、同伴回放或参考反馈的诊断信号压缩成简洁的自我生成苏格拉底提示和批评。此外，我们引入了因果信息增益（CIG），其具有不对称的ReLU门控阈值，用于将这些反思转换为稀疏、高精度的标记级优势调节。结合时间退火，这种机制在保持基础环境奖励的同时过滤掉分布噪声。在科学、数学和工具使用基准上的实验表明，AMR-SD显著优于现有基线，实现了稳健的长距离稳定性，并成功防止了晚期崩溃。
英文摘要：
The alignment of Large Language Models (LLMs) for complex reasoning heavily relies on Reinforcement Learning with Verifiable Rewards (RLVR). However, standard algorithms like GRPO apply sequence-level rewards uniformly to all tokens, creating a severe credit-assignment bottleneck. While on-policy self-distillation attempts to resolve this by conditioning a self-teacher on privileged contexts, direct exposure to raw oracle solutions often induces over-conditioned teacher distributions, implicit answer leakage, and late-stage training collapse. To overcome these limitations, we propose Asymmetric Meta-Reflective Self-Distillation (AMR-SD). Instead of conditioning directly on raw reference traces, AMR-SD inserts a reflection bottleneck: it compresses diagnostic signals -- from verifier outcomes, peer rollouts, or reference feedback -- into concise, self-generated Socratic hints and critiques. Furthermore, we introduce Causal Information Gain (CIG) with an asymmetric, ReLU-gated threshold to translate these reflections into sparse, highly precise token-level advantage modulations. Combined with temporal annealing, this mechanism preserves the base environmental reward while filtering out distributional noise. Experiments across scientific, mathematical, and tool-use benchmarks demonstrate that AMR-SD significantly outperforms existing baselines, achieving robust long-horizon stability and successfully preventing late-stage collapse.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
