Causal inference is essential for decision-making but remains challenging for non-experts. While large language models (LLMs) show promise in this domain, their precise causal estimation capabilities are still limited, and the impact of post-training on these abilities is insufficiently explored. This paper examines the extent to which post-training can enhance LLMs' capacity for causal inference. We introduce CauGym, a comprehensive dataset comprising seven core causal tasks for training and five diverse test sets. Using this dataset, we systematically evaluate five post-training approaches: SFT, DPO, KTO, PPO, and GRPO. Across five in-domain and four existing benchmarks, our experiments demonstrate that appropriate post-training enables smaller LLMs to perform causal inference competitively, often surpassing much larger models. Our 14B parameter model achieves 93.5% accuracy on the CaLM benchmark, compared to 55.4% by OpenAI o3. Furthermore, the post-trained LLMs exhibit strong generalization and robustness under real-world conditions such as distribution shifts and noisy data. Collectively, these findings provide the first systematic evidence that targeted post-training can produce reliable and robust LLM-based causal reasoners. Our data and GRPO-model are available at https://github.com/OpenCausaLab/CauGym.
Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
Intelligent Game and Decision Lab（智能游戏与决策实验室）；National University of Defense Technology（国防科技大学）；University of Melbourne（墨尔本大学）；University of Sydney（悉尼大学）；Hong Kong Baptist University（香港 Baptist 大学）
Center for Information and Language Processing (CIS)（信息与语言处理中心（CIS））；LMU Munich（慕尼黑大学）
How can pretrained language models (PLMs) learn factual knowledge from the training set? We investigate the two most important mechanisms: reasoning and memorization. Prior work has attempted to quantify the number of facts PLMs learn, but we present, using synthetic data, the first study that investigates the causal relation between facts present in training and facts learned by the PLM. For reasoning, we show that PLMs seem to learn to apply some symbolic reasoning rules correctly but struggle with others, including two-hop reasoning. Further analysis suggests that even the application of le…
Synergizing Large Language Models and Theory for Human-like Causal Reasoning
Center for Data Science, AAIS, Peking University（数据科学中心，AAIS，北京大学）；Wangxuan Institute of Computer Technology, Peking University（王轩计算机技术研究所，北京大学）；Tsinghua University（清华大学）；State Key Laboratory of General Artificial Intelligence（通用人工智能国家重点实验室）
CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists
Tsinghua University（清华大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Carnegie Mellon University（卡内基梅隆大学）；University of Chicago（芝加哥大学）；Adobe
Language Agents Meet Causality -- Bridging LLMs and Causal World Models
University of Amsterdam（阿姆斯特丹大学）；Institute for Logic, Language and Computation (ILLC)（逻辑、语言与计算研究所）；University of Edinburgh（爱丁堡大学）；Institute for Language, Cognition and Computation (ILCC)（语言、认知与计算研究所）
When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners
Harbin Institute of Technology（哈尔滨工业大学）；Singapore Management University（新加坡管理学院）；Monash University（墨尔本大学）；Singapore University of Technology and Design（新加坡科技设计大学）；National University of Singapore（国立新加坡大学）
Are Language Models Consequentialist or Deontological Moral Reasoners?
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Washington（华盛顿大学）；University of Zürich（苏黎世大学）；University of Michigan（密歇根大学）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
MemAudit: Post-hoc Auditing of Poisoned Agent Memory via Causal Attribution and Structural Anomaly Detection
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Qiyuan Tech（齐元科技）；Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机科学学院多媒体信息处理实验室）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）
City University of Hong Kong（香港城市大学）；Friedrich Miescher Institute for Biomedical Research（弗里德里希·米歇尔生物医学研究所）
Distributional representations of words, also known as word vectors, have become crucial for modern natural language processing tasks due to their wide applications. Recently, a growing body of word vector postprocessing algorithm has emerged, aiming to render off-the-shelf word vectors even stronger. In line with these investigations, we introduce a novel word vector postprocessing scheme under a causal inference framework. Concretely, the postprocessing pipeline is realized by Half-Sibling Regression (HSR), which allows us to identify and remove confounding noise contained in word vectors. C…
Thapar Institute of Engineering & Technology（塔帕尔工程技术学院）；The Chinese University of Hong Kong（香港中文大学）；University of Petroleum And Energy Studies（石油与能源研究大学）；Indian Institute of Technology, Patna（印度理工学院巴特那分校）；Amity University Rajasthan（拉贾斯坦阿米蒂大学）；Lakehead University（湖首大学）；University of Florida（佛罗里达大学）
Research community has witnessed substantial growth in the detection of mental health issues and their associated reasons from analysis of social media. We introduce a new dataset for Causal Analysis of Mental health issues in Social media posts (CAMS). Our contributions for causal analysis are two-fold: causal interpretation and causal categorization. We introduce an annotation schema for this task of causal analysis. We demonstrate the efficacy of our schema on two different datasets: (i) crawling and annotating 3155 Reddit posts and (ii) re-annotating the publicly available SDCNL dataset of…
Does adding a theorem to a paper affect its chance of acceptance? Does labeling a post with the author's gender affect the post popularity? This paper develops a method to estimate such causal effects from observational text data, adjusting for confounding features of the text such as the subject or writing quality. We assume that the text suffices for causal adjustment but that, in practice, it is prohibitively high-dimensional. To address this challenge, we develop causally sufficient embeddings, low-dimensional document representations that preserve sufficient information for causal identif…
Thapar Institute of Engineering & Technology（塔帕尔工程技术学院）；University of Florida（佛罗里达大学）；The Chinese University of Hong Kong（香港中文大学）
With recent developments in digitization of clinical psychology, NLP research community has revolutionized the field of mental health detection on social media. Existing research in mental health analysis revolves around the cross-sectional studies to classify users' intent on social media. For in-depth analysis, we investigate existing classifiers to solve the problem of causal categorization which suggests the inefficiency of learning based methods due to limited training samples. To handle this challenge, we use transformer models and demonstrate the efficacy of a pre-trained transfer learn…
