用于自适应推理与规划的强化上下文顺序恢复
Reinforced Context Order Recovery for Adaptive Reasoning and Planning
AI总结：
本文提出强化上下文顺序恢复（ReCOR）框架，利用强化学习和自监督 token 预测难度估计，从无标注文本中提取自适应生成顺序，在推理与规划任务上优于基线甚至真实顺序监督模型。
摘要：
现代因果语言模型，以及随后快速发展的离散扩散模型，如今能够生成各种各样有趣且有用的内容。然而，这些模型家族主要被训练为以固定（从左到右）或随机的顺序输出 token，这可能偏离 token 最初生成时的逻辑顺序。在本文中，我们观察到当前的因果模型和扩散模型在需要自适应 token 生成顺序才能可解地解决的问题上遇到困难，我们用 $\mathcal{V}$-information 框架刻画了这一现象。受此启发，我们提出了强化上下文顺序恢复（ReCOR），这是一个基于强化学习的框架，用于在没有标注的情况下从文本数据中提取自适应的、依赖数据的 token 生成顺序。ReCOR 通过 token 预测统计量进行自监督，估计预测每个未填充 token 的难度，并在训练和推理过程中自适应地选择下一个 token。在具有挑战性的推理和规划数据集上的实验表明，与基线相比，ReCOR 具有优越的性能，有时甚至优于使用真实顺序进行监督的 oracle 模型。
英文摘要：
Modern causal language models, followed by rapid developments in discrete diffusion models, can now produce a wide variety of interesting and useful content. However, these families of models are predominantly trained to output tokens with a fixed (left-to-right) or random order, which may deviate from the logical order in which tokens are generated originally. In this paper, we observe that current causal and diffusion models encounter difficulties in problems that require adaptive token generation orders to solve tractably, which we characterize with the $\mathcal{V}$-information framework. Motivated by this, we propose Reinforced Context Order Recovery (ReCOR), a reinforcement-learning-based framework to extract adaptive, data-dependent token generation orders from text data without annotations. Self-supervised by token prediction statistics, ReCOR estimates the hardness of predicting every unfilled token and adaptively selects the next token during both training and inference. Experiments on challenging reasoning and planning datasets demonstrate the superior performance of ReCOR compared with baselines, sometimes outperforming oracle models supervised with the ground-truth order.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
