A Survey on Metric Learning for Feature Vectors and Structured Data
University of Southern California（南加利福尼亚大学）；Laboratoire Hubert Curien UMR 5516（于贝尔·居里安实验室（UMR 5516））；Université de Saint-Etienne（圣艾蒂安大学）
Aurélien Bellet, Amaury Habrard, Marc Sebban
备注：
Technical report, 59 pages. Changes in v2: fixed typos and improved presentation. Changes in v3: fixed typos. Changes in v4: fixed typos and new methods
The need for appropriate ways to measure the distance or similarity between data is ubiquitous in machine learning, pattern recognition and data mining, but handcrafting such good metrics for specific problems is generally difficult. This has led to the emergence of metric learning, which aims at automatically learning a metric from data and has attracted a lot of interest in machine learning and related fields for the past ten years. This survey paper proposes a systematic review of the metric learning literature, highlighting the pros and cons of each approach. We pay particular attention to Mahalanobis distance metric learning, a well-studied and successful framework, but additionally present a wide range of methods that have recently emerged as powerful alternatives, including nonlinear metric learning, similarity learning and local metric learning. Recent trends and extensions, such as semi-supervised metric learning, metric learning for histogram data and the derivation of generalization guarantees, are also covered. Finally, this survey addresses metric learning for structured data, in particular edit distance learning, and attempts to give an overview of the remaining challenges in metric learning for the years to come.
Transformers flexibly operate over sets of real-valued vectors representing task-specific entities and their attributes, where each vector might encode one word-piece token and its position in a sequence, or some piece of information that carries no position at all. But as set processors, transformers are at a disadvantage in reasoning over more general graph-structured data where nodes represent entities and edges represent relations between entities. To address this shortcoming, we generalize transformer attention to consider and update edge vectors in each transformer layer. We evaluate thi…
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Koc University（科奇大学）
We introduce a Bayesian perspective for the structured matrix factorization problem. The proposed framework provides a probabilistic interpretation for existing geometric methods based on determinant minimization. We model input data vectors as linear transformations of latent vectors drawn from a distribution uniform over a particular domain reflecting structural assumptions, such as the probability simplex in Nonnegative Matrix Factorization and polytopes in Polytopic Matrix Factorization. We represent the rows of the linear transformation matrix as vectors generated independently from a nor…
Over the last several years, the field of Structured prediction in NLP has had seen huge advancements with sophisticated probabilistic graphical models, energy-based networks, and its combination with deep learning-based approaches. This survey provides a brief of major techniques in structured prediction and its applications in the NLP domains like parsing, sequence labeling, text generation, and sequence to sequence tasks. We also deep-dived into energy-based and attention-based techniques in structured prediction, identified some relevant open issues and gaps in the current state-of-the-art…
University of Southampton（南安普顿大学）；South China University of Technology（华南理工大学）
Supervised machine learning has several drawbacks that make it difficult to use in many situations. Drawbacks include: heavy reliance on massive training data, limited generalizability and poor expressiveness of high-level semantics. Low-shot Learning attempts to address these drawbacks. Low-shot learning allows the model to obtain good predictive power with very little or no training data, where structured knowledge plays a key role as a high-level semantic representation of human. This article will review the fundamental factors of low-shot learning technologies, with a focus on the operatio…
This survey presents a necessarily incomplete (and biased) overview of results at the intersection of arithmetic circuit complexity, structured matrices and deep learning. Recently there has been some research activity in replacing unstructured weight matrices in neural networks by structured ones (with the aim of reducing the size of the corresponding deep learning models). Most of this work has been experimental and in this survey, we formalize the research question and show how a recent work that combines arithmetic circuit complexity, structured matrices and deep learning essentially answe…
Towards Structured Deep Neural Network for Automatic Speech Recognition
National Taiwan University（台湾大学）
在本文中，我们提出了结构化深度神经网络（Structured DNN）作为一种结构化和深度学习算法，通过全局考虑结构之间的映射关系而不是逐项考虑结构化输入（如向量序列），学习找到最佳结构化对象（如标签序列）。当自动语音识别被视为这种结构化学习问题的特殊情况时，其中我们将声学矢量序列作为输入，并将音素标签序列作为输出，可以作为一个整体而不是逐帧地全面学习发音。结构化支持向量机（Structured Support Vector Machine，简称结构化SVM）是一种基于结构化学习的自动识别方法，但受到SVM线性性质的限制。在这里，我们提出了结构化DNN，以使用多层非线性变换作为结构化和深度学习算法。在TIMIT的初步实验中，它被证明优于结构化SVM。
Towards Structured Deep Neural Network for Automatic Speech Recognition
National Taiwan University（国立台湾大学）
在本文中，我们提出了结构化深度神经网络（结构化DNN）作为结构化和深度学习框架。这种方法可以通过全局地考虑结构之间的映射关系而不是逐项地考虑结构化输入（例如向量序列）来学习找到最佳结构化对象（例如标签序列）。当自动语音识别被视为这种结构化学习问题的特殊情况时，其中我们将声学矢量序列作为输入并且将音素标签序列作为输出，可以作为一个整体而不是逐帧地全面学习发音。结构化支持向量机（Structured Support Vector Machine，简称结构化SVM）是一种基于结构化学习的自动识别方法，但受到SVM线性性质的限制。在这里，我们提出了结构化DNN，以使用多层中的非线性变换作为结构化和深度学习方法。在TIMIT的初步实验中，这种方法被证明可以击败结构化SVM。
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；RWTH Aachen University（亚琛工业大学）
Accurate models of mechanical system dynamics are often critical for model-based control and reinforcement learning. Fully data-driven dynamics models promise to ease the process of modeling and analysis, but require considerable amounts of data for training and often do not generalize well to unseen parts of the state space. Combining data-driven modelling with prior analytical knowledge is an attractive alternative as the inclusion of structural knowledge into a regression model improves the model's data efficiency and physical integrity. In this article, we survey supervised regression mode…
IllinoisSL: A JAVA Library for Structured Prediction
Microsoft Research New England（微软新英格兰研究院）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Microsoft Research（微软研究院）；University of Utah（犹他大学）
Robert Bosch GmbH（罗伯特·博世有限公司）；Robert Bosch LLC（罗伯特·博世有限责任公司）
Automated driving is one of the most active research areas in computer science. Deep learning methods have made remarkable breakthroughs in machine learning in general and in automated driving (AD)in particular. However, there are still unsolved problems to guarantee reliability and safety of automated systems, especially to effectively incorporate all available information and knowledge in the driving task. Knowledge graphs (KG) have recently gained significant attention from both industry and academia for applications that benefit by exploiting structured, dynamic, and relational data. The c…
Predicting Census Survey Response Rates With Parsimonious Additive Models and Structured Interactions
Massachusetts Institute of Technology（麻省理工学院）；University of Sydney（悉尼大学）；United States Census Bureau（美国人口普查局）；MIT Sloan School of Management（麻省理工学院斯隆管理学院）
SILVA Networks as Structured Implicit Layers and Vector Attractors via Dynamic Interaction Fields
Federal University of Bahia（巴伊亚联邦大学）；Grupo de Estudos e Aplicação de Inteligência Artificial em Geofísica (GAIA), Federal University of Bahia（巴伊亚联邦大学地球物理学人工智能研究与应用组（GAIA））
Parallel Algorithms for Structured Sparse Support Vector Machines: Application in Music Genre Classification
Department of Statistics and Data Science, Southern University of Science and Technology（南方科技大学统计与数据科学系）；Faculty of Education, Languages, Psychology & Music, SEGi University（SEGi大学教育、语言、心理学与音乐学院）；College of Mathematics and Statistics, Chongqing University（重庆大学数学与统计学院）；School of Mathematical and Physical Sciences, Chongqing University of Science and Technology（重庆理工大学数学与物理科学学院）
Arizona State University（亚利桑那州立大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Graph neural networks, a powerful deep learning tool to model graph-structured data, have demonstrated remarkable performance on numerous graph learning tasks. To address the data noise and data scarcity issues in deep graph learning, the research on graph data augmentation has intensified lately. However, conventional data augmentation methods can hardly handle graph-structured data which is defined in non-Euclidean space with multi-modality. In this survey, we formally formulate the problem of graph data augmentation and further review the representative techniques and their applications in …
We present an expository overview of technical and cultural challenges to the development and adoption of automation at various stages in the data science prediction lifecycle, restricting focus to supervised learning with structured datasets. In addition, we review popular open source Python tools implementing common solution patterns for the automation challenges and highlight gaps where we feel progress still demands to be made.
