基于测地距离和上下文嵌入压缩的文本分类
On Geodesic Distances and Contextual Embedding Compression for Text Classification
摘要：
在一些内存受限的设置中，如物联网设备和网络数据管道，具有较小的上下文嵌入可能是有利的。我们调查的效果投影上下文嵌入数据（BERT）到一个流形上，并使用非线性降维技术来压缩这些嵌入。特别是，我们提出了一种新型的后处理方法，应用Isomap和PCA的组合。我们发现，测地线距离估计，估计黎曼流形上的最短路径，从Isomap的k-最近邻图支撑的压缩嵌入的性能，可与原来的BERT嵌入。在一个数据集上，我们发现，尽管降维了12倍，但在下游分类任务中，压缩嵌入的性能在原始BERT嵌入的0.1%以内。此外，我们发现，这种方法的效果特别好的任务依赖于句法数据，线性降维相比。这些结果表明，一种新的几何方法，以实现低维的文本嵌入从现有的Transformers和铺平道路的数据特定的和特定于应用程序的嵌入压缩的承诺。
英文摘要：
In some memory-constrained settings like IoT devices and over-the-network data pipelines, it can be advantageous to have smaller contextual embeddings. We investigate the efficacy of projecting contextual embedding data (BERT) onto a manifold, and using nonlinear dimensionality reduction techniques to compress these embeddings. In particular, we propose a novel post-processing approach, applying a combination of Isomap and PCA. We find that the geodesic distance estimations, estimates of the shortest path on a Riemannian manifold, from Isomap's k-Nearest Neighbors graph bolstered the performance of the compressed embeddings to be comparable to the original BERT embeddings. On one dataset, we find that despite a 12-fold dimensionality reduction, the compressed embeddings performed within 0.1% of the original BERT embeddings on a downstream classification task. In addition, we find that this approach works particularly well on tasks reliant on syntactic data, when compared with linear dimensionality reduction. These results show promise for a novel geometric approach to achieve lower dimensional text embeddings from existing transformers and pave the way for data-specific and application-specific embedding compressions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
