arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.21149cs.CL
收藏

从观察到干预:通过可执行因果干预提炼自生成上下文内演示

From Observation to Intervention: Distilling Self-Generated In-Context Demonstrations via Executable Causal Intervention

Shanghai Jiaotong University(上海交通大学);Academy of Military Science(军事科学院);National University of Defense Technology(国防科技大学)

Minming Ye, Min Liu, Haoang Chi, Ciheng Wu, Naiyang Guan

AI总结:

该研究针对现有ICL方法的观察等价性缺陷,提出ECI2D方法,通过可执行因果干预筛选演示,在推理基准上实现优于观察型ICL的性能。

摘要:

自生成上下文内学习(ICL)使大语言模型(LLM)能够合成自身的少样本演示,通常优于人工提示。然而,现有方法通过观察筛选演示,假设若理由能得出正确输出则为高质量。我们表明此类筛选存在“观察等价性”问题,即忠实因果理由与虚假事后理由均可产生相同输出,且仅通过观察无法在统计上区分二者。受独立因果机制启发,我们提出忠实理由隐含“迁移不变性”,即因果理由应在逻辑相似样本中保持有效。我们将该准则实例化为用于演示提炼的可执行因果干预(ECI2D),该方法将语言理由抽象为可执行程序,并在逻辑相似样本上应用因果干预(do算子),仅保留隐含迁移不变性的演示。在数学、逻辑和常识推理基准上的实验表明,ECI2D在保留远更少演示的同时,始终优于基于观察的ICL,且在测试分布偏移时提升超过7个百分点。

英文摘要:

Self-generated In-Context Learning (ICL) enables LLMs to synthesize their own few-shot demonstrations, often outperforming manual prompts. However, existing methods filter demonstrations by observation, assuming a rationale is high-quality if it leads to the correct output. We show such filtering suffers from \textit{observational equivalence}, whereby both faithful causal rationales and spurious post-hoc rationales can yield same outputs and are statistically indistinguishable from observation alone. Inspired by independent causal mechanisms, we argue that faithful rationales imply \textit{transfer invariance}, in which causal rationales should remain effective across logically similar samples. We instantiate this criterion as \textbf{E}xecutable \textbf{C}ausal \textbf{I}ntervention for \textbf{D}emonstrations \textbf{D}istillation (ECI2D), which abstracts language rationales into executable programs and applies the causal interventions ($do$-operator) on logically similar samples, retaining only those demonstrations that imply transfer invariance. Experiments on math, logical and commonsense reasoning benchmarks demonstrate that ECI2D consistently outperforms observation-based ICL while retaining far fewer demonstrations, with gains exceeding 7 points as the test distribution shifts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑