arXiv 1607.005342016-07-02cs.CL

Text comparison using word vector representations and dimensionality reduction

Aalto University（阿尔托大学）

Hendrik Heuer

英文摘要：

This paper describes a technique to compare large text sources using word vector representations (word2vec) and dimensionality reduction (t-SNE) and how it can be implemented using Python. The technique provides a bird's-eye view of text sources, e.g. text summaries and their source material, and enables users to explore text sources like a geographical map. Word vector representations capture many linguistic properties such as gender, tense, plurality and even semantic concepts like "capital city of". Using dimensionality reduction, a 2D map can be computed where semantically similar words are close to each other. The technique uses the word2vec model from the gensim Python library and t-SNE from scikit-learn.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

维度变化对词嵌入偏差的影响

Effect of dimensionality change on the bias of word embeddings

Indian Institute of Technology Guwahati（印度理工学院古瓦哈提分校）

本研究利用英文维基百科语料，考察维度变化对Word2Vec、fastText、ELMo和BERT四种词嵌入方法偏差的影响，发现偏差随维度显著变化且影响方式不统一，选择维度时应考虑这些因素。

2023-12-28cs.CL

Simple and Effective Dimensionality Reduction for Word Embeddings

Microsoft India（微软印度）

Word embeddings have become the basic building blocks for several natural language processing and information retrieval tasks. Pre-trained word embeddings are used in several downstream applications as well as for constructing representations for sentences, paragraphs and documents. Recently, there has been an emphasis on further improving the pre-trained word vectors through post-processing algorithms. One such area of improvement is the dimensionality reduction of the word embeddings. Reducing the size of word embeddings through dimensionality reduction can improve their utility in memory co…

2017-11-21cs.CL

The Role of Context Types and Dimensionality in Learning Word Embeddings

IBM Watson（IBM沃森）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）；Bar-Ilan University（巴伊兰大学）；Google（谷歌）

We provide the first extensive evaluation of how using different types of context to learn skip-gram word embeddings affects performance on a wide range of intrinsic and extrinsic NLP tasks. Our results suggest that while intrinsic tasks tend to exhibit a clear preference to particular types of contexts and higher dimensionality, more careful tuning is required for finding the optimal settings for most of the extrinsic tasks that we considered. Furthermore, for these extrinsic tasks, we find that once the benefit from increasing the embedding dimensionality is mostly exhausted, simple concaten…

2017-07-19cs.CL

Learning the Dimensionality of Word Embeddings

University of California, Irvine（加利福尼亚大学欧文分校）；Princeton University（普林斯顿大学）

We describe a method for learning word embeddings with data-dependent dimensionality. Our Stochastic Dimensionality Skip-Gram (SD-SG) and Stochastic Dimensionality Continuous Bag-of-Words (SD-CBOW) are nonparametric analogs of Mikolov et al.'s (2013) well-known 'word2vec' models. Vector dimensionality is made dynamic by employing techniques used by Cote & Larochelle (2016) to define an RBM with an infinite number of hidden units. We show qualitatively and quantitatively that SD-SG and SD-CBOW are competitive with their fixed-dimension counterparts while providing a distribution over embedding …

2017-04-13cs.CLcs.LGstat.ML

On the Dimensionality of Word Embedding

Stanford University（斯坦福大学）；Microsoft Corp.（微软公司）

In this paper, we provide a theoretical understanding of word embedding and its dimensionality. Motivated by the unitary-invariance of word embedding, we propose the Pairwise Inner Product (PIP) loss, a novel metric on the dissimilarity between word embeddings. Using techniques from matrix perturbation theory, we reveal a fundamental bias-variance trade-off in dimensionality selection for word embeddings. This bias-variance trade-off sheds light on many empirical observations which were previously unexplained, for example the existence of an optimal dimensionality. Moreover, new insights and d…

2018-12-11cs.CLcs.LGstat.ML

基于语言几何的无监督语义约简

Linguistic Geometries for Unsupervised Dimensionality Reduction

Georgia Institute of Technology（佐治亚理工学院）

文本文档是复杂的高维对象。为了有效地可视化这样的数据，重要的是降低其维数并将低维嵌入可视化为2-D或3-D散点图。在本文中，我们探讨降维方法，利用领域知识，以实现更好的低维嵌入和可视化的文件。我们考虑使用的几何形状手动指定的专家，自动从语料库统计的几何形状，从语言资源计算的几何形状。

2010-03-02cs.CL

更多相关论文收起更多论文

何时0.1%就足够了？分析降维和量化对文本嵌入压缩的联合效应

When Is 0.1% Enough? Analyzing the Combined Effects of Dimensionality Reduction and Quantization on Text Embedding Compression

Graduate School of Informatics, Nagoya University（名古屋大学信息学研究科）

本文系统研究了结合降维和量化方法压缩文本嵌入的效果，发现联合压缩可大幅减小嵌入尺寸（低至原始大小的0.1%）而几乎不损失性能，且最优策略因任务而异。

2026-10-21cs.CL

评估用于预训练句子嵌入的无监督降维方法

Evaluating Unsupervised Dimensionality Reduction Methods for Pretrained Sentence Embeddings

University of Liverpool（利物浦大学）

针对预训练语言模型句子嵌入高维性在资源受限设备上的问题，评估无监督降维方法，发现PCA等能降低近50%维度且不损失性能，甚至部分任务中进一步提升表现。

2024-03-20cs.CL

一种通过词嵌入进行主题建模的流程

A Process for Topic Modelling Via Word Embeddings

本文提出结合BERT词嵌入、UMAP降维与K-Means聚类的无监督主题建模流程，并通过TF-IDF、主题多样性和一致性验证其有效性。

2023-10-06cs.CL

Word2vec Skip-gram Dimensionality Selection via Sequential Normalized Maximum Likelihood

The University of Tokyo（东京大学）

In this paper, we propose a novel information criteria-based approach to select the dimensionality of the word2vec Skip-gram (SG). From the perspective of the probability theory, SG is considered as an implicit probability distribution estimation under the assumption that there exists a true contextual distribution among words. Therefore, we apply information criteria with the aim of selecting the best dimensionality so that the corresponding model can be as close as possible to the true distribution. We examine the following information criteria for the dimensionality selection problem: the A…

2020-08-25cs.CLcs.LGstat.ML

使用文档嵌入与降维检测文本数据中的协变量漂移

Detecting covariate drift in text data using document embeddings and dimensionality reduction

本文研究TF-IDF、Doc2Vec和BERT等文档嵌入结合LSA、PCA降维及KS、MMD检测方法在文本协变量漂移检测中的效果，发现特定组合表现更优，为提升文本分析模型可靠性提供依据。

2023-09-17cs.CLcs.LG

用于二维文本空间化的主题模型与降维方法的大规模评估

Large-Scale Evaluation of Topic Models and Dimensionality Reduction Methods for 2D Text Spatialization

University of Potsdam（波茨坦大学）；Hasso Plattner Institute（哈索·普拉特纳研究院）；Graz University of Technology（格拉茨技术大学）

本文针对二维文本空间化任务，大规模评估了多种主题模型与降维方法的组合，基于超过45,000个布局的量化指标，发现可解释的主题模型结合t-SNE降维能有效捕捉语料库结构并提升布局质量。

2023-07-17cs.CLcs.LG

Advancing the dimensionality reduction of speaker embeddings for speaker diarisation: disentangling noise and informing speech activity

Naver Corporation（NAVER公司）；Korea Advanced Institute of Science and Technology（韩国科学技术院）

The objective of this work is to train noise-robust speaker embeddings adapted for speaker diarisation. Speaker embeddings play a crucial role in the performance of diarisation systems, but they often capture spurious information such as noise, adversely affecting performance. Our previous work has proposed an auto-encoder-based dimensionality reduction module to help remove the redundant information. However, they do not explicitly separate such information and have also been found to be sensitive to hyper-parameter values. To this end, we propose two contributions to overcome these issues: (…

2022-11-03cs.CLcs.SD

Application of Fuzzy Clustering for Text Data Dimensionality Reduction

College of Information and Communications University of South Carolina（南卡罗来纳大学信息与通信学院）；Arnold School of Public Health at the University of South Carolina（南卡罗来纳大学阿诺德公共卫生学院）

Large textual corpora are often represented by the document-term frequency matrix whose elements are the frequency of terms; however, this matrix has two problems: sparsity and high dimensionality. Four dimension reduction strategies are used to address these problems. Of the four strategies, unsupervised feature transformation (UFT) is a popular and efficient strategy to map the terms to a new basis in the document-term frequency matrix. Although several UFT-based methods have been developed, fuzzy clustering has not been considered for dimensionality reduction. This research explores fuzzy c…

2019-09-21cs.CLcs.LGstat.AP

利用快速傅里叶变换在 Sentence Transformer 向量数据库中进行降维

Dimensionality Reduction in Sentence Transformer Vector Databases with Fast Fourier Transform

NextAI Systems（NextAI系统公司）

本文提出将 FFT 直接用于 Sentence Transformer 测试输入生成的嵌入向量降维，以提升 RAG、图像处理等场景的检索效率与可扩展性。

2024-04-09cs.AIcs.CLcs.DB