arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.15319cs.CLcs.AIstat.ML
收藏

大语言模型的因果性

Causality for Large Language Models

Zhejiang University(浙江大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Carnegie Mellon University(卡内基梅隆大学)

Anpeng Wu, Kun Kuang, Minqin Zhu, Yingrong Wang, Yujia Zheng, Kairong Han, Baohong Li, Guangyi Chen, Fei Wu, Kun Zhang

AI总结:

本文综述了如何将因果性融入大语言模型从词元嵌入、预训练到微调、对齐、推理和评估的全生命周期,以克服虚假相关导致的偏见与幻觉问题,并展望了六个未来研究方向。

摘要:

人工智能领域近期的突破推动了范式转变,拥有数十亿乃至数万亿参数的大语言模型(LLMs)在海量数据集上训练,在一系列语言任务中取得了前所未有的成功。然而,尽管取得了这些成功,LLMs 仍然依赖概率建模,这种建模往往捕捉的是根植于语言模式和社会刻板印象中的虚假相关性,而非实体与事件之间的真实因果关系。这一局限使 LLMs 容易受到人口统计学偏见、社会刻板印象和 LLM 幻觉等问题的影响。这些挑战凸显了将因果性融入 LLMs 的迫切需要,即超越相关性驱动的范式,构建更可靠且在伦理上更符合人类价值观的 AI 系统。虽然许多现有的综述和研究侧重于利用提示工程来激活 LLMs 的因果知识,或开发基准以评估其因果推理能力,但这些努力大多依赖人工干预来激活预训练模型。如何将因果性嵌入 LLMs 的训练过程,并构建更通用、更智能的模型,仍是一个尚未探索的问题。近期研究强调,LLMs 表现为“因果鹦鹉”,能够复述因果知识却并未真正理解或运用它。这些基于提示的方法仍局限于人工干预式的改进。本综述旨在填补这一空白,探索因果性如何在 LLMs 生命周期的每个阶段——从词元嵌入学习和基础模型训练到微调、对齐、推理和评估——增强 LLMs,为构建更具可解释性、更可靠且具备因果信息的模型铺平道路。此外,我们进一步概述了六个有前景的未来方向,以推进 LLM 发展、增强其因果推理能力,并解决这些模型当前面临的局限。

英文摘要:

Recent breakthroughs in artificial intelligence have driven a paradigm shift, where large language models (LLMs) with billions or trillions of parameters are trained on vast datasets, achieving unprecedented success across a series of language tasks. However, despite these successes, LLMs still rely on probabilistic modeling, which often captures spurious correlations rooted in linguistic patterns and social stereotypes, rather than the true causal relationships between entities and events. This limitation renders LLMs vulnerable to issues such as demographic biases, social stereotypes, and LLM hallucinations. These challenges highlight the urgent need to integrate causality into LLMs, moving beyond correlation-driven paradigms to build more reliable and ethically aligned AI systems. While many existing surveys and studies focus on utilizing prompt engineering to activate LLMs for causal knowledge or developing benchmarks to assess their causal reasoning abilities, most of these efforts rely on human intervention to activate pre-trained models. How to embed causality into the training process of LLMs and build more general and intelligent models remains unexplored. Recent research highlights that LLMs function as causal parrots, capable of reciting causal knowledge without truly understanding or applying it. These prompt-based methods are still limited to human interventional improvements. This survey aims to address this gap by exploring how causality can enhance LLMs at every stage of their lifecycle-from token embedding learning and foundation model training to fine-tuning, alignment, inference, and evaluation-paving the way for more interpretable, reliable, and causally-informed models. Additionally, we further outline six promising future directions to advance LLM development, enhance their causal reasoning capabilities, and address the current limitations these models face.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑