因果推理更青睐编码器：关于解码器-only模型极限的探讨
Causal Reasoning Favors Encoders: On The Limits of Decoder-Only Models
AI总结：
本文探讨了因果推理中编码器和编码器-解码器架构相较于仅解码器模型的优势，发现前者在多跳联合推理和分布偏移鲁棒性方面表现更优。
摘要：
在上下文学习（ICL）基础上，大型语言模型（LLMs）最近取得了进展，尽管其在因果推理中的作用和性能仍不明确。因果推理需要多跳组合和严格的联合控制，依赖于输入的虚假词汇关系可能会产生误导性结果。我们假设，由于它们能够将输入投影到潜在空间，编码器和编码器-解码器架构更适合这种多跳联合推理，而不是仅解码器模型。为此，我们比较了上述所有架构的微调版本在自然语言和非自然语言场景中的零样本和少样本ICL表现。我们发现，仅依靠ICL本身不足以实现可靠的因果推理，通常过于关注无关的输入特征。特别是，仅解码器模型对分布偏移明显脆弱，而微调的编码器和编码器-解码器模型在我们的测试中能够更稳健地泛化，包括非自然语言分割。两种架构仅在大规模情况下被仅解码器架构所匹配或超越。我们最后指出，对于成本效益高、短时间稳健的因果推理，经过针对性微调的编码器或编码器-解码器架构更受青睐。
英文摘要：
In context learning (ICL) underpins recent advances in large language models (LLMs), although its role and performance in causal reasoning remains unclear. Causal reasoning demands multihop composition and strict conjunctive control, and reliance on spurious lexical relations of the input could provide misleading results. We hypothesize that, due to their ability to project the input into a latent space, encoder and encoder decoder architectures are better suited for said multihop conjunctive reasoning versus decoder only models. To do this, we compare fine-tuned versions of all the aforementioned architectures with zero and few shot ICL in both natural language and non natural language scenarios. We find that ICL alone is insufficient for reliable causal reasoning, often overfocusing on irrelevant input features. In particular, decoder only models are noticeably brittle to distributional shifts, while finetuned encoder and encoder decoder models can generalize more robustly across our tests, including the non natural language split. Both architectures are only matched or surpassed by decoder only architectures at large scales. We conclude by noting that for cost effective, short horizon robust causal reasoning, encoder or encoder decoder architectures with targeted finetuning are preferable.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
