技能路径：从电路图中揭示语言技能
Skill Path: Unveiling Language Skills from Circuit Graphs
AI总结：
本文提出技能路径概念，通过分解、剪枝和因果中介三步框架从电路图中提取线性技能链，以克服原子消融和多余效应问题，并在三种语言技能上验证了其分层性与包容性。
摘要：
电路图发现已成为阐明语言模型技能机制的一种基础方法。尽管电路图在输出忠实性方面表现良好，但它们存在原子消融的问题，这会导致连接组件之间因果依赖关系的丢失。此外，其发现过程旨在保持输出忠实性，却无意中捕获了除孤立目标技能之外的多余效应。为缓解这些挑战，我们引入了技能路径，它通过在组件的线性链中隔离单个技能，提供了一种更精细、更紧凑的表示。为了实现从电路图中提取技能路径，我们提出了一个三步框架，包括分解、剪枝和剪枝后因果中介分析。特别地，我们提供了Transformer模型的完整线性分解，从而得到一个解耦的计算图。在剪枝之后，我们进一步采用因果分析技术，包括反事实和干预，从电路图中提取最终的技能路径。为强调技能路径的重要性，我们使用该框架研究了三种通用语言技能——前一个Token技能、归纳技能和上下文学习技能。实验支持了这些技能的两个关键性质，即分层性和包容性。
英文摘要：
Circuit graph discovery has emerged as a fundamental approach to elucidating the skill mechanistic of language models. Despite the output faithfulness of circuit graphs, they suffer from atomic ablation, which causes the loss of causal dependencies between connected components. In addition, their discovery process, designed to preserve output faithfulness, inadvertently captures extraneous effects other than an isolated target skill. To alleviate these challenges, we introduce skill paths, which offers a more refined and compact representation by isolating individual skills within a linear chain of components. To enable skill path extracting from circuit graphs, we propose a three-step framework, consisting of decomposition, pruning, and post-pruning causal mediation. In particular, we offer a complete linear decomposition of the transformer model which leads to a disentangled computation graph. After pruning, we further adopt causal analysis techniques, including counterfactuals and interventions, to extract the final skill paths from the circuit graph. To underscore the significance of skill paths, we investigate three generic language skills-Previous Token Skill, Induction Skill, and In-Context Learning Skill-using our framework. Experiments support two crucial properties of these skills, namely stratification and inclusiveness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
