Linguistic Geometries for Unsupervised Dimensionality Reduction

Text comparison using word vector representations and dimensionality reduction

This paper describes a technique to compare large text sources using word vector representations (word2vec) and dimensionality reduction (t-SNE) and how it can be implemented using Python. The technique provides a bird's-eye view of text sources, e.g. text summaries and their source material, and enables users to explore text sources like a geographical map. Word vector representations capture many linguistic properties such as gender, tense, plurality and even semantic concepts like "capital city of". Using dimensionality reduction, a 2D map can be computed where semantically similar words ar…

语言Transformer中高维抽象相的出现

Emergence of a High-Dimensional Abstraction Phase in Language Transformers

研究语言模型映射的几何特性与功能关系，在多个预训练Transformer和数据集中发现高内在维度的抽象相，该相与语言抽象、下游任务迁移及模型性能密切相关。

High-dimensional distributed semantic spaces for utterances

High-dimensional distributed semantic spaces have proven useful and effective for aggregating and processing visual, auditory, and lexical information for many tasks related to human-generated data. Human language makes use of a large and varying number of features, lexical and constructional items as well as contextual and discourse-specific data of various types, which all interact to represent various aspects of communicative information. Some of these features are mostly local and useful for the organisation of e.g. argument structure of a predication; others are persistent over the course…

邻居与亲属：语音嵌入如何反映世界各地的语言联系？

Neighbors and relatives: How do speech embeddings reflect linguistic connections across the world?

本文利用微调 XLS-R 模型的语音嵌入和 LDA 分析 106 种语言，证明嵌入距离可对应谱系、词汇与地理关系，为大规模语言类型学研究提供新方法。

Understanding and Improving Multi-Sense Word Embeddings via Extended Robust Principal Component Analysis

Unsupervised learned representations of polysemous words generate a large of pseudo multi senses since unsupervised methods are overly sensitive to contextual variations. In this paper, we address the pseudo multi-sense detection for word embeddings by dimensionality reduction of sense pairs. We propose a novel principal analysis method, termed Ex-RPCA, designed to detect both pseudo multi senses and real multi senses. With Ex-RPCA, we empirically show that pseudo multi senses are generated systematically in unsupervised method. Moreover, the multi-sense word embeddings can by improved by a si…

Exploring Word Embeddings for Unsupervised Textual User-Generated Content Normalization

Text normalization techniques based on rules, lexicons or supervised training requiring large corpora are not scalable nor domain interchangeable, and this makes them unsuitable for normalizing user-generated content (UGC). Current tools available for Brazilian Portuguese make use of such techniques. In this work we propose a technique based on distributed representation of words (or word embeddings). It generates continuous numeric vectors of high-dimensionality to represent words. The vectors explicitly encode many linguistic regularities and patterns, as well as syntactic and semantic word …