arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.09015cs.CLcs.LG
收藏

Luxical: 高速词典密集文本嵌入

Luxical: High-Speed Lexical-Dense Text Embeddings

DatologyAI
DatologyAI, :, Luke Merrick, Alex Fang, Aldo Carranza, Alvin Deng, Amro Abbas, Brett Larsen, Cody Blakeney, Darren Teh, David Schwab, Fan Pan, Haakon Mongstad, … 展开作者

DatologyAI, :, Luke Merrick, Alex Fang, Aldo Carranza, Alvin Deng, Amro Abbas, Brett Larsen, Cody Blakeney, Darren Teh, David Schwab, Fan Pan, Haakon Mongstad, Haoli Yin, Jack Urbanek, Jason Lee, Jason Telanoff, Josh Wills, Kaleigh Mentzer, Paul Burstein, Parth Doshi, Paul Burnstein, Pratyush Maini, Ricardo Monti, Rishabh Adiga, Scott Loftin, Siddharth Joshi, Spandan Das, Tony Jiang, Vineeth Dorna, Zhengping Wang, Bogdan Gaza, Ari Morcos, Matthew Leavitt

AI总结:

Luxical通过结合稀疏TF-IDF特征和知识蒸馏训练,实现高速词典密集文本嵌入,适用于大规模文本组织任务。

摘要:

前沿语言模型的质量越来越依赖我们组织大规模文本语料库的能力。当前主流工具在速度和灵活性之间进行权衡:词典分类器(如FastText)速度快但只能生成分类输出分数,而变换器文本嵌入模型的向量输出支持多种工作流程(如聚类、分类和检索)但生成成本高。我们介绍了Luxical,一个用于高速“词典密集”文本嵌入的库,旨在恢复两种方法的最佳特性以组织大规模文本。Luxical结合稀疏TF-IDF特征、一个小的ReLU网络和知识蒸馏训练计划,以极低的操作成本近似大型变换器嵌入模型。在本技术报告中,我们描述了Luxical的架构和训练目标,并在两个不同的应用中评估了一个具体的Luxical模型:一个针对网络爬虫文档检索测试和一个基于文本分类的端到端语言模型数据整理任务。在这些任务中,我们展示了比不同规模的神经基线模型快3倍到100倍的速度提升,并在数据整理任务中与FastText模型推理相当。在这些评估中,测试的Luxical模型在大规模文本组织中展示了有利的计算/质量权衡,与神经基线模型质量相当。Luxical作为开源软件可在https://github.com/datologyai/luxical上获得。

英文摘要:

Frontier language model quality increasingly hinges on our ability to organize web-scale text corpora for training. Today's dominant tools trade off speed and flexibility: lexical classifiers (e.g., FastText) are fast but limited to producing classification output scores, while the vector-valued outputs of transformer text embedding models flexibly support numerous workflows (e.g., clustering, classification, and retrieval) but are computationally expensive to produce. We introduce Luxical, a library for high-speed "lexical-dense" text embeddings that aims to recover the best properties of both approaches for web-scale text organization. Luxical combines sparse TF--IDF features, a small ReLU network, and a knowledge distillation training regimen to approximate large transformer embedding models at a fraction of their operational cost. In this technical report, we describe the Luxical architecture and training objective and evaluate a concrete Luxical model in two disparate applications: a targeted webcrawl document retrieval test and an end-to-end language model data curation task grounded in text classification. In these tasks we demonstrate speedups ranging from 3x to 100x over varying-sized neural baselines, and comparable to FastText model inference during the data curation task. On these evaluations, the tested Luxical model illustrates favorable compute/quality trade-offs for large-scale text organization, matching the quality of neural baselines. Luxical is available as open-source software at https://github.com/datologyai/luxical.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑