Shapley value is a classic notion from game theory, historically used to quantify the contributions of individuals within groups, and more recently applied to assign values to data points when training machine learning models. Despite its foundational role, a key limitation of the data Shapley framework is that it only provides valuations for points within a fixed data set. It does not account for statistical aspects of the data and does not give a way to reason about points outside the data set. To address these limitations, we propose a novel framework -- distributional Shapley -- where the value of a point is defined in the context of an underlying data distribution. We prove that distributional Shapley has several desirable statistical properties; for example, the values are stable under perturbations to the data points themselves and to the underlying data distribution. We leverage these properties to develop a new algorithm for estimating values from data, which comes with formal guarantees and runs two orders of magnitude faster than state-of-the-art algorithms for computing the (non-distributional) data Shapley values. We apply distributional Shapley to diverse data sets and demonstrate its utility in a data market setting.
TimeLAVA: Learning-Agnostic Valuation for Time Series Data
School of Mathematics and Statistics, The University of Melbourne（墨尔本大学数学与统计学学院）；Statistics, The University of Melbourne（墨尔本大学统计学系）；Statistics, University of Sydney（悉尼大学统计学系）；Responsible AI Research Centre, Australian Institute for Machine Learning（澳大利亚机器学习研究所负责任人工智能研究中心）；Amazon（亚马逊）；School of Mathematical Sciences, Adelaide University（阿德莱德大学数学科学学院）；Department of Machine Learning, MBZUAI（MBZUAI机器学习系）
The Shapley value (SV) has emerged as a promising method for data valuation. However, computing or estimating the SV is often computationally expensive. To overcome this challenge, Jia et al. (2019) propose an advanced SV estimation algorithm called ``Group Testing-based SV estimator'' which achieves favorable asymptotic sample complexity. In this technical note, we present several improvements in the analysis and design choices of this SV estimator. Moreover, we point out that the Group Testing-based SV estimator does not fully reuse the collected samples. Our analysis and insights contribute…
Data Shapley has recently been proposed as a principled framework to quantify the contribution of individual datum in machine learning. It can effectively identify helpful or harmful data points for a learning algorithm. In this paper, we propose Beta Shapley, which is a substantial generalization of Data Shapley. Beta Shapley arises naturally by relaxing the efficiency axiom of the Shapley value, which is not critical for machine learning settings. Beta Shapley unifies several popular data valuation methods and includes data Shapley as a special case. Moreover, we prove that Beta Shapley has …
Variance reduced Shapley value estimation for trustworthy data valuation
Virginia Tech（弗吉尼亚理工大学）；China Aerospace Academy of Systems Science and Engineering（中国航天系统科学与工程研究院）；Southern University of Science and Technology（南方科技大学）
Quantifying the value of data is a fundamental problem in machine learning. Data valuation has multiple important use cases: (1) building insights about the learning task, (2) domain adaptation, (3) corrupted sample discovery, and (4) robust learning. To adaptively learn data values jointly with the target task predictor model, we propose a meta learning framework which we name Data Valuation using Reinforcement Learning (DVRL). We employ a data value estimator (modeled by a deep neural network) to learn how likely each datum is used in training of the predictor model. We train the data value …
University of California at Berkeley（加州大学伯克利分校）；ETH, Zurich（苏黎世联邦理工学院）；Zhejiang University（浙江大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
"How much is my data worth?" is an increasingly common question posed by organizations and individuals alike. An answer to this question could allow, for instance, fairly distributing profits among multiple data contributors and determining prospective compensation when data breaches happen. In this paper, we study the problem of data valuation by utilizing the Shapley value, a popular notion of value which originated in cooperative game theory. The Shapley value defines a unique payoff scheme that satisfies many desiderata for the notion of data value. However, the Shapley value often require…
Data Valuation by Fusing Global and Local Statistical Information
Peking University（北京大学）；University of Chinese Academy of Sciences（中国科学院大学）；Hong Kong Baptist University（香港 Baptist大学）；Renmin University of China（中国人民大学）
As data becomes the fuel driving technological and economic growth, a fundamental challenge is how to quantify the value of data in algorithmic predictions and decisions. For example, in healthcare and consumer markets, it has been suggested that individuals should be compensated for the data that they generate, but it is not clear what is an equitable valuation for individual data. In this work, we develop a principled framework to address data valuation in the context of supervised machine learning. Given a learning algorithm trained on $n$ data points to produce a predictor, we propose data…
We propose to align distributional data from the perspective of Wasserstein means. We raise the problem of regularizing Wasserstein means and propose several terms tailored to tackle different problems. Our formulation is based on the variational transportation to distribute a sparse discrete measure into the target domain. The resulting sparse representation well captures the desired property of the domain while reducing the mapping cost. We demonstrate the scalability and robustness of our method with examples in domain adaptation, point set registration, and skeleton layout.
GMValuator: Similarity-based Data Valuation for Generative Models
University of British Columbia（不列颠哥伦比亚大学）；University of Washington（华盛顿大学）；Princeton University（普林斯顿大学）；Stanford University（斯坦福大学）；Vector Institute（矢量研究院）
Error Estimate and Convergence Analysis for Data Valuation
National Center for Applied Mathematics, Tianjin University（应用数学国家中心，天津大学）；School of Mechanical and Aerospace Engineering, Jilin University（机械与航空航天工程学院，吉林大学）；School of Mathematics, Physics and Computing, University of Southern Queensland（数学、物理与计算学院，南方昆士兰大学）
University of Virginia（弗吉尼亚大学）；University of Chicago（芝加哥大学）
Data valuation, or the valuation of individual datum contributions, has seen growing interest in machine learning due to its demonstrable efficacy for tasks such as noisy label detection. In particular, due to the desirable axiomatic properties, several Shapley value approximation methods have been proposed. In these methods, the value function is typically defined as the predictive accuracy over the entire development set. However, this limits the ability to differentiate between training instances that are helpful or harmful to their own classes. Intuitively, instances that harm their own cl…
Data valuation and subset selection have emerged as valuable tools for application-specific selection of important training data. However, the efficiency-accuracy tradeoffs of state-of-the-art methods hinder their widespread application to many AI workflows. In this paper, we propose a novel 2-phase solution to this problem. Phase 1 selects representative checkpoints from an SGD-like training algorithm, which are used in phase-2 to estimate the approximate training data values, e.g. decrease in validation loss due to each training point. A key contribution of this paper is CheckSel, an Orthogo…
Real estate appraisal is a complex and important task, that can be made more precise and faster with the help of automated valuation tools. Usually the value of some property is determined by taking into account both structural and geographical characteristics. However, while geographical information is easily found, obtaining significant structural information requires the intervention of a real estate expert, a professional appraiser. In this paper we propose a Web data acquisition methodology, and a Machine Learning model, that can be used to automatically evaluate real estate properties. T…
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
Many recent works on understanding deep learning try to quantify how much individual data instances influence the optimization and generalization of a model. Such attempts reveal characteristics and importance of individual instances, which may provide useful information in diagnosing and improving deep learning. However, most of the existing works on data valuation require actual training of a model, which often demands high-computational cost. In this paper, we provide a training-free data valuation score, called complexity-gap score, which is a data-centric score to quantify the influence o…
Feedzai（Feedzai公司）；Instituto Superior Técnico, Universidade de Lisboa（里斯本大学高等技术学院）；Instituto de Telecomunicações（电信研究所）
Data valuation is a ML field that studies the value of training instances towards a given predictive task. Although data bias is one of the main sources of downstream model unfairness, previous work in data valuation does not consider how training instances may influence both performance and fairness of ML models. Thus, we propose Fairness-Aware Data vauatiOn (FADO), a data valuation framework that can be used to incorporate fairness concerns into a series of ML-related tasks (e.g., data pre-processing, exploratory data analysis, active learning). We propose an entropy-based data valuation met…
The Shapley value (SV) and Least core (LC) are classic methods in cooperative game theory for cost/profit sharing problems. Both methods have recently been proposed as a principled solution for data valuation tasks, i.e., quantifying the contribution of individual datum in machine learning. However, both SV and LC suffer computational challenges due to the need for retraining models on combinatorially many data subsets. In this work, we propose to boost the efficiency in computing Shapley value or Least core by learning to estimate the performance of a learning algorithm on unseen data combina…
Valuation problems, such as feature interpretation, data valuation and model valuation for ensembles, become increasingly more important in many machine learning applications. Such problems are commonly solved by well-known game-theoretic criteria, such as Shapley value or Banzhaf value. In this work, we present a novel energy-based treatment for cooperative games, with a theoretical justification by the maximum entropy framework. Surprisingly, by conducting variational inference of the energy-based model, we recover various game-theoretic valuation criteria through conducting one-step fixed p…
Temporal-Decay Shapley: A Time-Aware Data Valuation Framework for Time-Series Data
School of Artificial Intelligence, Guangzhou University（广州大学人工智能学院）；School of Public Finance and Taxation, Guangdong University of Finance and Economics（广东财经大学公共财政与税收学院）
We discuss a data market technique based on intrinsic (relevance and uniqueness) as well as extrinsic value (influenced by supply and demand) of data. For intrinsic value, we explain how to perform valuation of data in absolute terms (i.e just by itself), or relatively (i.e in comparison to multiple datasets) or in conditional terms (i.e valuating new data given currently existing data).
National University of Singapore（新加坡国立大学）；Carnegie Mellon University（卡内基梅隆大学）；Institute for Infocomm Research（信息通信研究院）；Agency for Science, Technology and Research（科技研究局）
Beijing Advanced Innovation Center for Big Data and Brain Computing, Beihang University（北京航空航天大学大数据与脑机智能高精尖创新中心）；School of Computing, University of Leeds（利兹大学计算学院）；UrBrain Technology；Department of Computer Science, University of Illinois at Chicago（伊利诺伊大学芝加哥分校计算机科学系）；Department of Computer Science and Engineering, Lehigh University（理海大学计算机科学与工程系）
We present Luce, the first life-long predictive model for automated property valuation. Luce addresses two critical issues of property valuation: the lack of recent sold prices and the sparsity of house data. It is designed to operate on a limited volume of recent house transaction data. As a departure from prior work, Luce organizes the house data in a heterogeneous information network (HIN) where graph nodes are house entities and attributes that are important for house price valuation. We employ a Graph Convolutional Network (GCN) to extract the spatial information from the HIN for house-re…
EcoVal: An Efficient Data Valuation Framework for Machine Learning
RespAI Lab（RespAI实验室）；KIIT Bhubaneswar（布巴内斯瓦尔KIIT学院）；NUS Business School（新加坡国立大学商学院）；National University of Singapore（新加坡国立大学）；Adam Smith Business School（亚当·斯密商学院）；University of Glasgow（格拉斯哥大学）；School of Computing（计算机学院）
Swiss Institute for Information Science（瑞士信息科学研究所）；University of Applied Sciences of the Grisons（格劳宾登应用科学大学）
Performing company valuations within the domain of biotechnology, pharmacy and medical technology is a challenging task, especially when considering the unique set of risks biotech start-ups face when entering new markets. Companies specialized in global valuation services, therefore, combine valuation models and past experience with heterogeneous metrics and indicators that provide insights into a company's performance. This paper illustrates how automated knowledge discovery, extraction and data fusion can be used to (i) obtain additional indicators that provide insights into the success of …
Lightweight Time Series Data Valuation on Time Series Foundation Models via In-Context Finetuning
Sun Yat-sen University State Key Laboratory of Internet Architecture（中山大学互联网架构国家重点实验室）；Tsinghua University（清华大学）；National University of Singapore（新加坡国立大学）
