学习可组合思维链
Learning Composable Chains-of-Thought
AI总结：
该研究针对无标注思维链数据的组合推理任务，提出通过可组合思维链训练原子模型并结合多任务学习或模型合并，搭配拒绝采样微调，提升了大型语言模型在组合推理任务上的泛化性能。
摘要：
训练大型语言模型（LLM）进行推理的常用方法是在分布内推理问题的思维链（CoT）轨迹上进行训练，但针对每个感兴趣的问题获取此类标注数据成本高昂。我们希望推理模型能泛化到训练分布之外，理想情况下是组合式泛化：结合原子推理技能来解决更困难的、未见过的推理任务。在处理没有标注CoT数据的目标组合任务时，我们朝着推理技能的组合式泛化迈出了一步。我们发现，仅在原子任务的CoT数据上训练模型会导致泛化能力有限，但对组成原子任务的CoT格式进行最小限度的修改使其可组合，可带来性能提升。我们可以使用可组合CoT数据在原子任务上训练“原子CoT”模型，并结合多任务学习或模型合并，以在目标组合任务上获得更好的零样本性能。这种组合模型可进一步使用拒绝采样微调（RFT），在少量组合数据上进行自举训练。在字符串操作和自然语言技能组合任务上的结果表明，在给定的训练数据预算内，在可组合CoT上训练LLM的性能优于多任务学习和持续微调基线方法。
英文摘要：
A common approach for teaching large language models (LLMs) to reason is to train on chain-of-thought (CoT) traces of in-distribution reasoning problems, but such annotated data is costly to obtain for every problem of interest. We want reasoning models to generalize beyond their training distribution, and ideally to generalize compositionally: combine atomic reasoning skills to solve harder, unseen reasoning tasks. We take a step towards compositional generalization of reasoning skills when addressing a target compositional task that has no labeled CoT data. We find that simply training models on CoT data of atomic tasks leads to limited generalization, but minimally modifying CoT formats of constituent atomic tasks to be composable can lead to improvements. We can train "atomic CoT" models on the atomic tasks with Composable CoT data and combine them with multitask learning or model merging for better zero-shot performance on the target compositional task. Such a combined model can be further bootstrapped on a small amount of compositional data using rejection sampling fine-tuning (RFT). Results on string operations and natural language skill compositions show that training LLMs on Composable CoT outperforms multitask learning and continued fine-tuning baselines within a given training data budget.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
