We present a framework for analysing agent incentives using causal influence diagrams. We establish that a well-known criterion for value of information is complete. We propose a new graphical criterion for value of control, establishing its soundness and completeness. We also introduce two new concepts for incentive analysis: response incentives indicate which changes in the environment affect an optimal decision, while instrumental control incentives establish whether an agent can influence its utility via a variable X. For both new concepts, we provide sound and complete graphical criteria. We show by example how these results can help with evaluating the safety and fairness of an AI system.
Agents are systems that optimize an objective function in an environment. Together, the goal and the environment induce secondary objectives, incentives. Modeling the agent-environment interaction using causal influence diagrams, we can answer two fundamental questions about an agent's incentives directly from the graph: (1) which nodes can the agent have an incentivize to observe, and (2) which nodes can the agent have an incentivize to control? The answers tell us which information and influence points need extra protection. For example, we may want a classifier for job applications to not u…
We present a general framework for training safe agents whose naive incentives are unsafe. As an example, manipulative or deceptive behaviour can improve rewards but should be avoided. Most approaches fail here: agents maximize expected return by any means necessary. We formally describe settings with 'delicate' parts of the state which should not be used as a means to an end. We then train agents to maximize the causal effect of actions on the expected return which is not mediated by the delicate parts of state, using Causal Influence Diagram analysis. The resulting agents have no incentive t…
Revisiting Few-Shot Learning from a Causal Perspective
School of Computer Science and Engineering, Sun Yat-Sen University(中山大学计算机科学与工程学院);School of Artificial Intelligence, Sun Yat-Sen University(中山大学人工智能学院)
This paper presents a topological learning-theoretic perspective on causal inference by introducing a series of topologies defined on general spaces of structural causal models (SCMs). As an illustration of the framework we prove a topological causal hierarchy theorem, showing that substantive assumption-free causal inference is possible only in a meager set of SCMs. Thanks to a known correspondence between open sets in the weak topology and statistically verifiable hypotheses, our results show that inductive assumptions sufficient to license valid causal inferences are statistically unverifia…
Can humans get arbitrarily capable reinforcement learning (RL) agents to do their bidding? Or will sufficiently capable RL agents always find ways to bypass their intended objectives by shortcutting their reward signal? This question impacts how far RL can be scaled, and whether alternative paradigms must be developed in order to build safe artificial general intelligence. In this paper, we study when an RL agent has an instrumental goal to tamper with its reward process, and describe design principles that prevent instrumental goals for two different types of reward tampering (reward function…
With the increasing pervasive use of machine learning in social and economic settings, there has been an interest in the notion of machine bias in the AI community. Models trained on historic data reflect biases that exist in society and propagated them to the future through their decisions. There are three prominent metrics of machine fairness used in the community, and it has been shown statistically that it is impossible to satisfy them all at the same time. This has led to an ambiguity with regards to the definition of fairness. In this report, a causal perspective to the impossibility the…
Are Heterogeneous Graph Neural Networks Truly Effective for Node Classification? A Causal Perspective
College of Computing and Data Science, Nanyang Technological University(计算与数据科学学院,南洋理工大学);Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly (LILY), Nanyang Technological University(老年人积极生活卓越研究中心(LILY),南洋理工大学);Alibaba-NTU Singapore Joint Research Institute (ANGEL), Nanyang Technological University(阿里巴巴-南洋理工大学新加坡联合研究机构(ANGEL),南洋理工大学)
Recent years have witnessed the rapid growth of machine learning in a wide range of fields such as image recognition, text classification, credit scoring prediction, recommendation system, etc. In spite of their great performance in different sectors, researchers still concern about the mechanism under any machine learning (ML) techniques that are inherently black-box and becoming more complex to achieve higher accuracy. Therefore, interpreting machine learning model is currently a mainstream topic in the research community. However, the traditional interpretable machine learning focuses on th…
Model identifiability is a desirable property in the context of unsupervised representation learning. In absence thereof, different models may be observationally indistinguishable while yielding representations that are nontrivially related to one another, thus making the recovery of a ground truth generative model fundamentally impossible, as often shown through suitably constructed counterexamples. In this note, we discuss one such construction, illustrating a potential failure case of an identifiability result presented in "Desiderata for Representation Learning: A Causal Perspective" by Wa…
A Shift in Perspective on Causality in Domain Generalization
CHAI Hub(CHAI中心);The University of Edinburgh(爱丁堡大学);University of Manchester(曼彻斯特大学);Archimedes, Athena Research Center(阿奇米德与雅典研究所);King's College London(伦敦国王学院);University College London(伦敦大学学院);National and Kapodistrian University of Athens(雅典国家与卡波迪斯蒂亚纳大学)
Systems with Switching Causal Relations: A Meta-Causal Perspective
Technical University of Darmstadt(达姆施塔特工业大学);Hessian Center for AI (hessian.AI)(黑森州人工智能中心);Eindhoven University of Technology(埃因霍温理工大学);Centre for Cognitive Science(认知科学中心);German Research Center for AI (DFKI)(德国人工智能研究中心)
Training datasets for machine learning often have some form of missingness. For example, to learn a model for deciding whom to give a loan, the available training data includes individuals who were given a loan in the past, but not those who were not. This missingness, if ignored, nullifies any fairness guarantee of the training procedure when the model is deployed. Using causal graphs, we characterize the missingness mechanisms in different real-world scenarios. We show conditions under which various distributions, used in popular fairness algorithms, can or can not be recovered from the trai…
Copenhagen Causality Lab and Pioneer Centre for AI, University of Copenhagen, Denmark(哥本哈根因果实验室和先锋人工智能中心,哥本哈根大学,丹麦);Cooperative AI Foundation, United Kingdom(协作人工智能基金会,英国)
LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification
Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);National Key Laboratory of Space Integrated Information System(空间信息集成系统国家重点实验室);University of Chinese Academy of Sciences(中国科学院大学);Tsinghua University(清华大学)
Identifying and controlling bias is a key problem in empirical sciences. Causal diagram theory provides graphical criteria for deciding whether and how causal effects can be identified from observed (nonexperimental) data by covariate adjustment. Here we prove equivalences between existing as well as new criteria for adjustment and we provide a new simplified but still equivalent notion of d-separation. These lead to efficient algorithms for two important tasks in causal diagram analysis: (1) listing minimal covariate adjustments (with polynomial delay); and (2) identifying the subdiagram invo…
National University of Singapore(新加坡国立大学);Hong Kong University of Science and Technology(香港科技大学);Microsoft Research(微软研究院)
Membership inference (MI) attacks highlight a privacy weakness in present stochastic training methods for neural networks. It is not well understood, however, why they arise. Are they a natural consequence of imperfect generalization only? Which underlying causes should we address during training to mitigate these attacks? Towards answering such questions, we propose the first approach to explain MI attacks and their connection to generalization based on principled causal reasoning. We offer causal graphs that quantitatively explain the observed MI attack performance achieved for $6$ attack va…
A novel approach to the relationships between data features -- based on comprehensive examination of mathematical, technological, and causal methodology
Legal Research Institute of Korea University(高丽大学法律研究所);Human-Inspired AI Research, Korea University(高丽大学类人人工智能研究院)
Rethinking Dimensional Rationale in Graph Contrastive Learning from Causal Perspective
Science & Technology on Integrated Information System Laboratory, Institute of Software Chinese Academy of Sciences(中国科学院软件研究所综合信息系统技术国家级重点实验室);State Key Laboratory of Intelligent Game(智能博弈国家重点实验室);University of Chinese Academy of Sciences(中国科学院大学);State Key Laboratory of Computer Science, Institute of Software Chinese Academy of Sciences(中国科学院软件研究所计算机科学国家重点实验室)
Learning to Allocate Incentives for Incentivized Advertising via Offline Model-Based Reinforcement Learning
State Key Laboratory of Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);ByteDance(字节跳动);The Hong Kong University of Science and Technology(香港科技大学)
ProPublica's analysis of recidivism predictions produced by Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) software tool for the task, has shown that the predictions were racially biased against African American defendants. We analyze the COMPAS data using a causal reformulation of the underlying algorithmic fairness problem. Specifically, we assess whether COMPAS exhibits racial bias against African American defendants using FACT, a recently introduced causality grounded measure of algorithmic fairness. We use the Neyman-Rubin potential outcomes framework for ca…
Contextual bandit and reinforcement learning algorithms have been successfully used in various interactive learning systems such as online advertising, recommender systems, and dynamic pricing. However, they have yet to be widely adopted in high-stakes application domains, such as healthcare. One reason may be that existing approaches assume that the underlying mechanisms are static in the sense that they do not change over different environments. In many real-world systems, however, the mechanisms are subject to shifts across environments which may invalidate the static environment assumption…
The Importance of Causality in Decision Making: A Perspective on Recommender Systems
Free University of Bozen-Bolzano(波尔扎诺自由大学);University of Milano-Bicocca(米兰比可卡大学);F. Hoffmann - La Roche Ltd(F. Hoffmann - 罗氏公司);University of Klagenfurt(克拉根福大学)
École Polytechnique Fédérale de Lausanne (EPFL)(洛桑联邦理工学院)
Learning behavioral patterns from observational data has been a de-facto approach to motion forecasting. Yet, the current paradigm suffers from two shortcomings: brittle under distribution shifts and inefficient for knowledge transfer. In this work, we propose to address these challenges from a causal representation perspective. We first introduce a causal formalism of motion forecasting, which casts the problem as a dynamic process with three groups of latent variables, namely invariant variables, style confounders, and spurious features. We then introduce a learning framework that treats eac…
The Frame Problem (FP) is a puzzle in philosophy of mind and epistemology, articulated by the Stanford Encyclopedia of Philosophy as follows: "How do we account for our apparent ability to make decisions on the basis only of what is relevant to an ongoing situation without having explicitly to consider all that is not relevant?" In this work, we focus on the causal variant of the FP, the Causal Frame Problem (CFP). Assuming that a reasoner's mental causal model can be (implicitly) represented by a causal Bayes net, we first introduce a notion called Potential Level (PL). PL, in essence, encode…