CoT-ICL Lab：一个用于研究上下文演示中思维链学习的合成框架
CoT-ICL Lab: A Synthetic Framework for Studying Chain-of-Thought Learning from In-Context Demonstrations
AI总结：
CoT-ICL Lab通过合成数据集解耦因果结构与处理函数，研究思维链上下文学习，发现深度对CoT关键，更多示例助浅层模型匹敌深层性能。
摘要：
我们介绍了CoT-ICL Lab，一个用于生成合成标记化数据集并系统研究语言模型中思维链（CoT）上下文学习（ICL）的框架和方法论。CoT-ICL Lab通过解耦（1）链标记生成中涉及的因果结构与（2）底层标记处理函数，实现对上下文示例复杂度的精细控制。我们在这些数据集上训练仅解码器变压器（参数高达7亿），并表明CoT在不同模型规模下加速了准确率向更高值的转变。特别地，我们发现模型深度对于在有限上下文示例下利用CoT至关重要，而更多示例有助于浅层模型匹配更深层模型的性能。此外，在训练过程中限制标记处理函数的多样性，通过ICL改善了因果结构学习。我们还通过分析变压器嵌入和注意力图来解读这些转变。总体而言，CoT-ICL Lab为语言模型中ICL和CoT的理论与实证见解提供了一个简单而强大的测试平台。
英文摘要：
We introduce CoT-ICL Lab, a framework and methodology to generate synthetic tokenized datasets and systematically study chain-of-thought (CoT) in-context learning (ICL) in language models. CoT-ICL Lab allows fine grained control over the complexity of in-context examples by decoupling (1) the causal structure involved in chain token generation from (2) the underlying token processing functions. We train decoder-only transformers (up to 700M parameters) on these datasets and show that CoT accelerates the accuracy transition to higher values across model sizes. In particular, we find that model depth is crucial for leveraging CoT with limited in-context examples, while more examples help shallow models match deeper model performance. Additionally, limiting the diversity of token processing functions throughout training improves causal structure learning via ICL. We also interpret these transitions by analyzing transformer embeddings and attention maps. Overall, CoT-ICL Lab serves as a simple yet powerful testbed for theoretical and empirical insights into ICL and CoT in language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
