针对结构化数据共享中隐私保护与机器学习效用难以兼顾的问题，提出多目标优化匿名化模型，在降低信息损失和攻击风险的同时保持可比的 ML 性能。
摘要：
各组织正在收集海量数据，但它们往往缺乏充分提取洞见所需的能力。因此，它们越来越多地与分析师或研究人员等外部专家共享数据，以从中获取价值。然而，这种做法会带来重大的隐私风险。为解决数据共享中的隐私问题，已有多种技术被提出。但是，这些方法通常会降低数据效用，影响机器学习（ML）模型的性能。我们的研究识别了现有隐私保护优化模型中的关键局限，尤其是在处理分类变量以及跨多样化数据集评估有效性方面。我们提出了一种新颖的多目标优化模型，可同时最小化信息损失并最大化抵御攻击的保护能力。该模型使用多样化数据集进行了实证验证，并与两种现有算法进行了比较。我们评估了信息损失、遭受链接攻击或同质性攻击的个体数量，以及匿名化后的 ML 性能。结果表明，我们的模型实现了更低的信息损失，并更有效地缓解了攻击风险；在某些情况下，与替代算法相比，减少了易受这些攻击影响的个体数量。此外，相对于原始数据或由其他方法匿名化的数据，我们的模型保持了可比的 ML 性能。我们的发现凸显了隐私保护和 ML 模型性能方面的显著改进，为在数据共享中平衡隐私与效用提供了一个全面且可扩展的框架。
英文摘要：
Organizations are collecting vast amounts of data, but they often lack the capabilities needed to fully extract insights. As a result, they increasingly share data with external experts, such as analysts or researchers, to gain value from it. However, this practice introduces significant privacy risks. Various techniques have been proposed to address privacy concerns in data sharing. However, these methods often degrade data utility, impacting the performance of machine learning (ML) models. Our research identifies key limitations in existing optimization models for privacy preservation, particularly in handling categorical variables, and evaluating effectiveness across diverse datasets. We propose a novel multi-objective optimization model that simultaneously minimizes information loss and maximizes protection against attacks. This model is empirically validated using diverse datasets and compared with two existing algorithms. We assess information loss, the number of individuals subject to linkage or homogeneity attacks, and ML performance after anonymization. The results indicate that our model achieves lower information loss and more effectively mitigates the risk of attacks, reducing the number of individuals susceptible to these attacks compared to alternative algorithms in some cases. Additionally, our model maintains comparable ML performance relative to the original data or data anonymized by other methods. Our findings highlight significant improvements in privacy protection and ML model performance, offering a comprehensive and extensible framework for balancing privacy and utility in data sharing.
DiffAnon: Diffusion-based Prosody Control for Voice Anonymization
Center for Language and Speech Processing, Johns Hopkins University, USA（语言与语音处理中心，约翰霍普金斯大学，美国）；Human Language Technology Center of Excellence (COE), Johns Hopkins University, USA（人类语言技术卓越中心（COE），约翰霍普金斯大学，美国）
Traditional approaches for data anonymization consider relational data and textual data independently. We propose rx-anon, an anonymization approach for heterogeneous semi-structured documents composed of relational and textual attributes. We map sensitive terms extracted from the text to the structured data. This allows us to use concepts like k-anonymity to generate a joined, privacy-preserved version of the heterogeneous data input. We introduce the concept of redundant sensitive information to consistently anonymize the heterogeneous data. To control the influence of anonymization over uns…
Queen Mary University of London（伦敦玛丽女王大学）；Imperial College London（帝国理工学院）
Motion sensors such as accelerometers and gyroscopes measure the instant acceleration and rotation of a device, in three dimensions. Raw data streams from motion sensors embedded in portable and wearable devices may reveal private information about users without their awareness. For example, motion data might disclose the weight or gender of a user, or enable their re-identification. To address this problem, we propose an on-device transformation of sensor data to be shared for specific applications, such as monitoring selected daily activities, without revealing information that enables user …
AI-Driven Anonymization: Protecting Personal Data Privacy While Leveraging Machine Learning
Sam Houston State University（山姆休斯顿州立大学）；San Fransisco Bay University（旧金山湾大学）；Sun Yat-Sen University（中山大学）；University of Chicago（芝加哥大学）；Guangdong University of Technology（广东工业大学）
In intractable, undirected graphical models, an intuitive way of creating structured mean field approximations is to select an acyclic tractable subgraph. We show that the hardness of computing the objective function and gradient of the mean field objective qualitatively depends on a simple graph property. If the tractable subgraph has this property- we call such subgraphs v-acyclic-a very fast block coordinate ascent algorithm is possible. If not, optimization is harder, but we show a new algorithm based on the construction of an auxiliary exponential family that can be used to make inference…
NICTA（澳大利亚国家信息与通信技术研究院）；The Australian National University（澳大利亚国立大学）
A successful approach to structured learning is to write the learning objective as a joint function of linear parameters and inference messages, and iterate between updates to each. This paper observes that if the inference problem is "smoothed" through the addition of entropy terms, for fixed messages, the learning objective reduces to a traditional (non-structured) logistic regression problem with respect to parameters. In these logistic regression problems, each training example has a bias term determined by the current set of messages. Based on this insight, the structured energy function …
Benchmarking Time Series Generation Methods for Privacy-Preserving Forecasting
University of Minho（米尼奥大学）；Universidade do Porto（波尔图大学）；Laboratory for Artificial Intelligence and Computer Science (LIACC)（人工智能与计算机科学实验室（LIACC））；Fraunhofer Portugal AICOS（弗劳恩霍夫葡萄牙AICOS机构）；INESCTEC；University of Coimbra（科英布拉大学）
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Massachusetts Institute of Technology（麻省理工学院）；UC Berkeley（加州大学伯克利分校）
The use of min-max optimization in adversarial training of deep neural network classifiers and training of generative adversarial networks has motivated the study of nonconvex-nonconcave optimization objectives, which frequently arise in these applications. Unfortunately, recent results have established that even approximate first-order stationary points of such objectives are intractable, even under smoothness conditions, motivating the study of min-max objectives with additional structure. We introduce a new class of structured nonconvex-nonconcave min-max optimization problems, proposing a …
Heidelberg University（海德堡大学）；Tufts University（塔夫茨大学）；Amazon Development Center（亚马逊开发中心）
Stochastic structured prediction under bandit feedback follows a learning protocol where on each of a sequence of iterations, the learner receives an input, predicts an output structure, and receives partial feedback in form of a task loss evaluation of the predicted structure. We present applications of this learning scenario to convex and non-convex objectives for structured prediction and analyze them as stochastic first-order methods. We present an experimental evaluation on problems of natural language processing over exponential output spaces, and compare convergence speed across differe…
Speeding Up Multi-Objective Hyperparameter Optimization by Task Similarity-Based Meta-Learning for the Tree-Structured Parzen Estimator
Department of Computer Science, University of Freiburg, Germany（弗赖堡大学计算机科学系）；Artificial Intelligence Research Center, AIST, Tokyo, Japan（日本科学技术厅人工智能研究中心）
McMaster University（麦克马斯特大学）；University of Toronto（多伦多大学）
An Objective Structured Practical Examination (OSPE) is an effective and robust, but resource-intensive, means of evaluating anatomical knowledge. Since most OSPEs employ short answer or fill-in-the-blank style questions, the format requires many people familiar with the content to mark the exams. However, the increasing prevalence of online delivery for anatomy and physiology courses could result in students losing the OSPE practice that they would receive in face-to-face learning sessions. The purpose of this study was to test the accuracy of Decision Trees (DTs) in marking OSPE questions as…
Northeastern University（东北大学）；University of Oxford（牛津大学）；Alan Turing Institute（艾伦·图灵研究所）
Deep latent-variable models learn representations of high-dimensional data in an unsupervised manner. A number of recent efforts have focused on learning representations that disentangle statistically independent axes of variation by introducing modifications to the standard objective function. These approaches generally assume a simple diagonal Gaussian prior and as a result are not able to reliably disentangle discrete factors of variation. We propose a two-level hierarchical objective to control relative degree of statistical independence between blocks of variables and individual variables…
Rutgers University（罗格斯大学）；Toyota Technological Institute at Chicago（芝加哥丰田工业大学）
We propose learning discrete structured representations from unlabeled data by maximizing the mutual information between a structured latent variable and a target variable. Calculating mutual information is intractable in this setting. Our key technical contribution is an adversarial objective that can be used to tractably estimate mutual information assuming only the feasibility of cross entropy calculation. We develop a concrete realization of this general formulation with Markov distributions over binary encodings. We report critical and unexpected findings on practical aspects of the objec…
One Loss to Rule Them All: Marked Time-to-Event for Structured EHR Foundation Models
Department of Computer Science, Columbia University（哥伦比亚大学计算机科学系）；Department of Biomedical Informatics, Columbia University（哥伦比亚大学生物医学信息学系）；Department of Computational Medicine, UCLA（洛杉矶大学计算医学系）；Formation Bio
We present a framework to train a structured prediction model by performing smoothing on the inference algorithm it builds upon. Smoothing overcomes the non-smoothness inherent to the maximum margin structured prediction objective, and paves the way for the use of fast primal gradient-based optimization algorithms. We illustrate the proposed framework by developing a novel primal incremental optimization algorithm for the structural support vector machine. The proposed algorithm blends an extrapolation scheme for acceleration and an adaptive smoothing scheme and builds upon the stochastic vari…
ShanghaiTech University（上海科技大学）；Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences（中国科学院上海微系统与信息技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）
Knowledge distillation is a critical technique to transfer knowledge between models, typically from a large model (the teacher) to a more fine-grained one (the student). The objective function of knowledge distillation is typically the cross-entropy between the teacher and the student's output distributions. However, for structured prediction problems, the output space is exponential in size; therefore, the cross-entropy objective becomes intractable to compute and optimize directly. In this paper, we derive a factorized form of the knowledge distillation objective for structured prediction, w…
ML-based approach to classification and generation of structured light propagation in turbulent media
Division of Mathematical Sciences, Nanyang Technological University（南洋理工大学数学科学系）；Committee on Computational and Applied Mathematics, University of Chicago（芝加哥大学计算与应用数学委员会）；Departments of Statistics and Mathematics, University of Chicago（芝加哥大学统计学与数学系）
Simon Fraser University（西蒙弗雷泽大学）；Microsoft Research（微软研究院）
We propose a randomized nonmonotone block proximal gradient (RNBPG) method for minimizing the sum of a smooth (possibly nonconvex) function and a block-separable (possibly nonconvex nonsmooth) function. At each iteration, this method randomly picks a block according to any prescribed probability distribution and solves typically several associated proximal subproblems that usually have a closed-form solution, until a certain progress on objective value is achieved. In contrast to the usual randomized block coordinate descent method [23,20], our method has a nonmonotone flavor and uses variable…
The Australian National University（澳大利亚国立大学）；NICTA（澳大利亚国家信息与通信技术研究院）；INRIA（法国国家信息与自动化研究所）
A thesis submitted for the degree of Doctor of Philosophy of The Australian National University. In this work we introduce several new optimisation methods for problems in machine learning. Our algorithms broadly fall into two categories: optimisation of finite sums and of graph structured objectives. The finite sum problem is simply the minimisation of objective functions that are naturally expressed as a summation over a large number of terms, where each term has a similar or identical weight. Such objectives most often appear in machine learning in the empirical risk minimisation framework …
Pruning is a promising approach to compress deep learning models in order to deploy them on resource-constrained edge devices. However, many existing pruning solutions are based on unstructured pruning, which yields models that cannot efficiently run on commodity hardware; and they often require users to manually explore and tune the pruning process, which is time-consuming and often leads to sub-optimal results. To address these limitations, this paper presents Automatic Attention Pruning (AAP), an adaptive, attention-based, structured pruning approach to automatically generate small, accurat…
Many predicted structured objects (e.g., sequences, matchings, trees) are evaluated using the F-score, alignment error rate (AER), or other multivariate performance measures. Since inductively optimizing these measures using training data is typically computationally difficult, empirical risk minimization of surrogate losses is employed, using, e.g., the hinge loss for (structured) support vector machines. These approximations often introduce a mismatch between the learner's objective and the desired application performance, leading to inconsistency. We take a different approach: adversarially…
Stochastic zeroth-order (SZO), or gradient-free, optimization allows to optimize arbitrary functions by relying only on function evaluations under parameter perturbations, however, the iteration complexity of SZO methods suffers a factor proportional to the dimensionality of the perturbed function. We show that in scenarios with natural sparsity patterns as in structured prediction applications, this factor can be reduced to the expected number of active features over input-output pairs. We give a general proof that applies sparse SZO optimization to Lipschitz-continuous, nonconvex, stochastic…
Graduate School of Information Science and Electrical Engineering, Kyushu University（九州大学信息科学与电气工程研究生院）；Faculty of Information Science and Electrical Engineering, Kyushu University（九州大学信息科学与电气工程学院）；Institute of Mathematics and Computer Science, University of São Paulo（圣保罗大学数学与计算机科学研究所）
Structured evolutionary algorithms have been investigated for some time. However, they have been under-explored specially in the field of multi-objective optimization. Despite their good results, the use of complex dynamics and structures make their understanding and adoption rate low. Here, we propose the general subpopulation framework that has the capability of integrating optimization algorithms without restrictions as well as aid the design of structured algorithms. The proposed framework is capable of generalizing most of the structured evolutionary algorithms, such as cellular algorithm…
Department of XXX, University of YYY, Location, Country；School of ZZZ, Institute of WWW, Location, Country
Assembly of multi-part physical structures is both a valuable end product for autonomous robotics, as well as a valuable diagnostic task for open-ended training of embodied intelligent agents. We introduce a naturalistic physics-based environment with a set of connectable magnet blocks inspired by children's toy kits. The objective is to assemble blocks into a succession of target blueprints. Despite the simplicity of this objective, the compositional nature of building diverse blueprints from a set of blocks leads to an explosion of complexity in structures that agents encounter. Furthermore,…
Online Learning with Multiple Fairness Regularizers via Graph-Structured Feedback
Department of Mathematics, National University of Singapore（新加坡国立大学数学系）；Department of Computer Science, Czech Technical University（捷克技术大学计算机科学系）；Dyson School of Design Engineering, Imperial College London（伦敦帝国理工学院设计工程戴森学院）；Imperial College London（伦敦帝国理工学院）
We present a fully-supervized method for learning to segment data structured by an adjacency graph. We introduce the graph-structured contrastive loss, a loss function structured by a ground truth segmentation. It promotes learning vertex embeddings which are homogeneous within desired segments, and have high contrast at their interface. Thus, computing a piecewise-constant approximation of such embeddings produces a graph-partition close to the objective segmentation. This loss is fully backpropagable, which allows us to learn vertex embeddings with deep learning algorithms. We evaluate our m…
University of California, Davis（加州大学戴维斯分校）；Carnegie Mellon University（卡内基梅隆大学）
Text style transfer aims to modify the style of a sentence while keeping its content unchanged. Recent style transfer systems often fail to faithfully preserve the content after changing the style. This paper proposes a structured content preserving model that leverages linguistic information in the structured fine-grained supervisions to better preserve the style-independent content during style transfer. In particular, we achieve the goal by devising rich model objectives based on both the sentence's lexical information and a language model that conditions on content. The resulting model the…
