通过循环对齐推理增强自回归思维链
Enhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoning
AI总结：
针对自回归思维链生成长且正确轨迹的挑战，该研究提出RELAY方法，将思维链推理步骤与循环Transformer的迭代对齐并施加中间监督，提升其泛化性以生成超长推理链，进而微调自回归模型并显著提升其性能。
摘要：
思维链（CoT）提示已成为增强语言模型推理能力的强大技术。然而，生成长且正确的CoT轨迹具有挑战性。近期研究表明，循环Transformer（Looped Transformers）具备卓越的长度泛化能力，但其有限的通用性和适应性使其无法作为自回归解决方案的替代方案。为更好地利用循环Transformer的优势，我们提出RELAY（通过循环对齐迭代推理）。具体而言，我们将思维链（CoT）推理步骤与循环迭代对齐，并在循环Transformer训练期间应用中间监督。这种额外的迭代级监督不仅保留了循环Transformer的长度泛化能力，还使其能够预测未见过数据的CoT推理步骤。因此，我们利用该循环Transformer为超出训练长度的复杂问题生成准确的推理链，随后用于微调自回归模型。我们进行了大量实验，结果证明了我们方法的有效性，自回归模型的性能得到显著提升。代码将在https://github.com/qifanyu/RELAY发布。
英文摘要：
Chain-of-Thought (CoT) prompting has emerged as a powerful technique for enhancing language model's reasoning capabilities. However, generating long and correct CoT trajectories is challenging. Recent studies have demonstrated that Looped Transformers possess remarkable length generalization capabilities, but their limited generality and adaptability prevent them from serving as an alternative to auto-regressive solutions. To better leverage the strengths of Looped Transformers, we propose RELAY (REasoning through Loop Alignment iterativelY). Specifically, we align the steps of Chain-of-Thought (CoT) reasoning with loop iterations and apply intermediate supervision during the training of Looped Transformers. This additional iteration-wise supervision not only preserves the Looped Transformer's ability for length generalization but also enables it to predict CoT reasoning steps for unseen data. Therefore, we leverage this Looped Transformer to generate accurate reasoning chains for complex problems that exceed the training length, which will then be used to fine-tune an auto-regressive model. We conduct extensive experiments, and the results demonstrate the effectiveness of our approach, with significant improvements in the performance of the auto-regressive model. Code will be released at https://github.com/qifanyu/RELAY.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
