About compression of vocabulary in computer oriented languages

PoC：通过性能预测实现面向性能的上下文压缩

PoC: Performance-oriented Context Compression for Large Language Models via Performance Prediction

本文提出PoC方法，通过设定性能底线而非压缩比来优化大语言模型的上下文压缩，设计两种预测器并验证其在问答和摘要任务中的性能优势。

Extremely Small BERT Models from Mixed-Vocabulary Training

Pretrained language models like BERT have achieved good results on NLP tasks, but are impractical on resource-limited devices due to memory footprint. A large fraction of this footprint comes from the input embeddings with large input vocabulary and embedding dimensions. Existing knowledge distillation methods used for model compression cannot be directly applied to train student models with reduced vocabulary sizes. To this end, we propose a distillation method to align the teacher and student embeddings via mixed-vocabulary training. Our method compresses BERT-LARGE to a task-agnostic model …

通过像素级回退克服词汇表约束

Overcoming Vocabulary Constraints with Pixel-level Fallback

提出一种无词汇表编码器，通过将文本渲染为像素生成输入嵌入来增强预训练语言模型，显著提升了机器翻译和跨语言迁移性能，无需大量重训即可增强多语能力并降低解码延迟。

潜在核心分词器：有意义地进行压缩

Latent Core Tokenizer: Compress, but Meaningfully

该研究提出与语言无关的潜在核心分词器（LCT），通过分离结构发现与词汇构建，在104种语言的分词任务中优于BPE等方法，多语言下游基准得分提升，证明压缩 alone 无法预测表示质量。

动态规划引导的分层BPE与词汇剪枝实证分析

Dynamic-Programming-Guided Hierarchical BPE and Empirical Analysis of Vocabulary Pruning

提出DH-BPE方法，结合动态规划与分层依赖进行词汇剪枝，在固定词汇预算下提升压缩率，并在多个规模上优于标准BPE等基线。

贪心最长匹配分词的联合优化

Joint Optimization for Greedy Longest-match Tokenization

研究针对贪心最长匹配解码，提出联合优化方法JOLT，将词汇学习设为整数规划，通过贪心一致性约束等确保训练与部署时分词对齐，求解线性规划松弛扩大优化规模，实验表明其能缩小与最优压缩差距，减少令牌数量。

Bayesian Compression for Natural Language Processing

In natural language processing, a lot of the tasks are successfully solved with recurrent neural networks, but such models have a huge number of parameters. The majority of these parameters are often concentrated in the embedding layer, which size grows proportionally to the vocabulary length. We propose a Bayesian sparsification technique for RNNs which allows compressing the RNN dozens or hundreds of times without time-consuming hyperparameters tuning. We also generalize the model for vocabulary sparsification to filter out unnecessary words and compress the RNN even further. We show that th…

为何纠结于连续潜变量？通过渲染压缩实现可解释的离散潜变量推理

Why Struggle with Continuous Latents? Interpretable Discrete Latent Reasoning via Rendered Compression

提出离散潜变量推理（DLR）方法，将连续潜状态转化为离散令牌，通过渲染压缩实现高效、可解释的潜空间推理，在五个基准上优于先前方法并达到20倍压缩。

每种语言有其分词器：面向高效多语言大语言模型的模块化分词器

To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs

针对多语言大模型共享词汇表导致压缩不均、内存占用高和推理慢的问题，提出模块化分词器框架，通过可提取子分词器及采样预训练策略，实现高效训练与推理，不牺牲性能。

分词不止于压缩

Tokenization Is More Than Compression

该研究提出PathPiece分词器检验“token越少下游性能越好”的假设，发现假设不成立，进而分析分词三阶段的设计因素，训练并公开了64个不同参数规模的语言模型，为分词器设计提供新见解。

最小化基于预训练语言模型（PLM）的少样本意图检测器

Minimizing PLM-Based Few-Shot Intent Detectors

针对PLM基少样本意图检测器体积大、难在资源受限环境部署的问题，结合LLM数据增强、知识蒸馏与V-Prune词表剪枝，实现21倍内存压缩且性能基本不变。

基于信息压缩的轻量级概念字典学习用于文本分类

Lightweight Conceptual Dictionary Learning for Text Classification Using Information Compression

本文提出一种基于LZW压缩和标签互信息精炼的轻量级监督字典学习框架，用于文本分类，在有限词汇数据集上以约10%参数量接近顶尖模型性能，并引入IPAR度量评估信息论表现。

GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking

Model compression is essential for serving large deep neural nets on devices with limited resources or applications that require real-time responses. As a case study, a state-of-the-art neural language model usually consists of one or more recurrent layers sandwiched between an embedding layer used for representing input tokens and a softmax layer for generating output tokens. For problems with a very large vocabulary size, the embedding and the softmax matrices can account for more than half of the model size. For instance, the bigLSTM model achieves state-of- the-art performance on the One-B…

FR-Spec：通过频率排序的投机采样加速大词表语言模型

FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling

FR-Spec提出频率排序的投机采样框架，通过将起草搜索限制在频率优先的token子集来压缩词表空间，在保证输出分布等价的前提下将LM Head计算开销降低75%，相对EAGLE-2平均加速1.12倍。

解释与缓解跨语言分词器的不平等现象

Explaining and Mitigating Crosslingual Tokenizer Inequities

本研究通过训练约7000个单语分词器，发现词汇表大小和预分词显著影响跨语言token溢价，并可通过确定最优词汇表大小或使用超词分词器来缓解该不平等现象。

TokAlign++: 通过更好的词 token 对齐推进词汇适应

TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment

TokAlign++通过学习更好的词 token 对齐词典，提升多语言模型的词汇适应性能，实验显示其能提升多语言文本压缩率并保持模型能力，仅需1k步即可恢复基础模型性能。

A Batch Noise Contrastive Estimation Approach for Training Large Vocabulary Language Models

Training large vocabulary Neural Network Language Models (NNLMs) is a difficult task due to the explicit requirement of the output layer normalization, which typically involves the evaluation of the full softmax function over the complete vocabulary. This paper proposes a Batch Noise Contrastive Estimation (B-NCE) approach to alleviate this problem. This is achieved by reducing the vocabulary, at each time step, to the target words in the batch and then replacing the softmax by the noise contrastive estimation approach, where these words play the role of targets and noise samples at the same t…

大语言模型分词中的计数与最小成本编码

Counting and Min-Cost Encoding for Tokenization in Large Language Models

提出CNF分词器训练与MCE最小成本编码算法，通过全局优化分割和计数过滤构建词表，相比BPE在压缩率、词元效率和词表利用率上显著提升，且下游性能相当。

Knowledge Distillation of Russian Language Models with Reduction of Vocabulary

Today, transformer language models serve as a core component for majority of natural language processing tasks. Industrial application of such models requires minimization of computation time and memory footprint. Knowledge distillation is one of approaches to address this goal. Existing methods in this field are mainly focused on reducing the number of layers or dimension of embeddings/hidden representations. Alternative option is to reduce the number of tokens in vocabulary and therefore the embeddings matrix of the student model. The main problem with vocabulary minimization is mismatch bet…

面向低资源语言的多语言编码器语言模型压缩研究

On Multilingual Encoder Language Model Compression for Low-Resource Languages

本文提出一种结合知识蒸馏、结构化剪枝、截断和词表裁剪的极致压缩方法，将多语言编码器模型压缩为低资源语言的单语言小模型，在保持竞争力的同时实现高达92%的压缩率。

Extreme Model Compression for On-device Natural Language Understanding

In this paper, we propose and experiment with techniques for extreme compression of neural natural language understanding (NLU) models, making them suitable for execution on resource-constrained devices. We propose a task-aware, end-to-end compression approach that performs word-embedding compression jointly with NLU task learning. We show our results on a large-scale, commercial NLU system trained on a varied set of intents with huge vocabulary sizes. Our approach outperforms a range of baselines and achieves a compression rate of 97.4% with less than 3.7% degradation in predictive performanc…

泰米尔语的字素感知印度语分词器：大规模训练与内在评估

A Grapheme-Aware Indic Tokenizer for Tamil: Large-Scale Training and Intrinsic Evaluation

本文提出一种针对泰米尔语的字素感知分词器，通过可逆Unicode映射保留完整字素簇，在压缩比和碎片化等内在指标上优于五种主流多语言分词器。

中间层和标签真的有必要吗？一种通用语言模型蒸馏方法

Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Method

针对现有知识蒸馏依赖中间层、金标且忽略词表参数的问题，提出无需中间层和金标、含词表压缩的通用语言模型蒸馏方法，在SuperGLUE上超越25种SOTA方法，平均得分高3%。

BrahmicTokenizer-131K：一种可替代o200k_base的印度文字兼容分词器

BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base

提出BrahmicTokenizer-131K，一种131072词汇量的字节级BPE分词器，通过两阶段改造在保持非印度文字性能的同时，显著提升印度文字的压缩效率。

分析子词分词的认知合理性

Analyzing Cognitive Plausibility of Subword Tokenization

提出评估子词分词认知合理性的新范式，通过分析分词器输出与人类词汇决策任务表现的相关性，发现UnigramLM算法的认知合理性及派生语素覆盖较差。

Tensorized Embedding Layers for Efficient Model Compression

The embedding layers transforming input words into real vectors are the key components of deep neural networks used in natural language processing. However, when the vocabulary is large, the corresponding weight matrices can be enormous, which precludes their deployment in a limited resource setting. We introduce a novel way of parametrizing embedding layers based on the Tensor Train (TT) decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance. We evaluate our method on a wide range of benchmarks in natural language …

zip2zip++：面向推理时的最小质量损失的令牌压缩

zip2zip++: Towards Inference-Time Token Compression with Minimal Quality Loss

zip2zip++通过改进压缩层与LZW编解码器，解决了动态词汇表压缩的质量损失问题，在不修改预训练语言模型的情况下，缩小了zip2zip与未压缩基线的质量差距，提升了困惑度与数学推理准确率。

AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search

Large pre-trained language models such as BERT have shown their effectiveness in various natural language processing tasks. However, the huge parameter size makes them difficult to be deployed in real-time applications that require quick inference with limited resources. Existing methods compress BERT into small models while such compression is task-independent, i.e., the same compressed BERT for all different downstream tasks. Motivated by the necessity and benefits of task-oriented BERT compression, we propose a novel compression method, AdaBERT, that leverages differentiable Neural Architec…

Adaptive Natural Language Generation for Task-oriented Dialogue via Reinforcement Learning

When a natural language generation (NLG) component is implemented in a real-world task-oriented dialogue system, it is necessary to generate not only natural utterances as learned on training data but also utterances adapted to the dialogue environment (e.g., noise from environmental sounds) and the user (e.g., users with low levels of understanding ability). Inspired by recent advances in reinforcement learning (RL) for language generation tasks, we propose ANTOR, a method for Adaptive Natural language generation for Task-Oriented dialogue via Reinforcement learning. In ANTOR, a natural langu…

Compression of Recurrent Neural Networks for Efficient Language Modeling

Recurrent neural networks have proved to be an effective method for statistical language modeling. However, in practice their memory and run-time complexity are usually too large to be implemented in real-time offline mobile applications. In this paper we consider several compression techniques for recurrent neural networks including Long-Short Term Memory models. We make particular attention to the high-dimensional output problem caused by the very large vocabulary size. We focus on effective compression methods in the context of their exploitation on devices: pruning, quantization, and matri…

Building Task-Oriented Visual Dialog Systems Through Alternative Optimization Between Dialog Policy and Language Generation

Reinforcement learning (RL) is an effective approach to learn an optimal dialog policy for task-oriented visual dialog systems. A common practice is to apply RL on a neural sequence-to-sequence (seq2seq) framework with the action space being the output vocabulary in the decoder. However, it is difficult to design a reward function that can achieve a balance between learning an effective policy and generating a natural dialog response. This paper proposes a novel framework that alternatively trains a RL policy for image guessing and a supervised seq2seq model to improve dialog generation qualit…

Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decomposition

Word-embeddings are vital components of Natural Language Processing (NLP) models and have been extensively explored. However, they consume a lot of memory which poses a challenge for edge deployment. Embedding matrices, typically, contain most of the parameters for language models and about a third for machine translation systems. In this paper, we propose Distilled Embedding, an (input/output) embedding compression method based on low-rank matrix decomposition and knowledge distillation. First, we initialize the weights of our decomposed matrices by learning to reconstruct the full pre-traine…

Brain-CLIPLM: 用于EEG到文本解码的语义压缩

Brain-CLIPLM: Semantic Compression for EEG-to-Text Decoding

该研究提出Brain-CLIPLM框架，通过语义锚点恢复和锚点引导的句子重建，解决EEG信号低信噪比和信息带宽限制的问题，实现了更高的文本检索准确率。

World-to-Words：通过视觉-语言模型中的快速映射实现基于真实世界的开放词汇习得

World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models

本文提出GOVA框架和OctoBERT模型，通过在图像-文本对上以接地为目标的预训练，验证视觉-语言模型能更快速稳健地学习新词汇，揭示了接地能力对新词学习的自举作用。

面向开放词汇分割的可迁移且有理论依据的效率优化方法

Transferable and Principled Efficiency for Open-Vocabulary Segmentation

针对开放词汇分割现有方法计算开销大、传统效率优化方案依赖启发式规则难以迁移的问题，提出有理论依据的可迁移效率优化策略，用小模型实现了更优的精度-成本权衡。

Self-Normalized Importance Sampling for Neural Language Modeling

To mitigate the problem of having to traverse over the full vocabulary in the softmax normalization of a neural language model, sampling-based training criteria are proposed and investigated in the context of large vocabulary word-based neural language models. These training criteria typically enjoy the benefit of faster training and testing, at a cost of slightly degraded performance in terms of perplexity and almost no visible drop in word error rate. While noise contrastive estimation is one of the most popular choices, recently we show that other sampling-based criteria can also perform we…

SupraTok：用于提升语言模型性能的跨边界分词

SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance

针对语言建模中分词受空格边界限制的问题，提出SupraTok分词器，通过跨空格边界的模块化设计实现压缩率、训练速度及下游任务性能的提升。

面向任务的小型语言模型加速

Task-Centric Acceleration of Small-Language Models

本文提出TASC框架，通过任务自适应序列压缩方法提升小型语言模型的推理效率，同时保持任务性能。

基于频率顺序的标记化以提升文本压缩

Frequency-Ordered Tokenization for Better Text Compression

基于频率顺序的标记化通过优化词汇表结构提升文本压缩效率，适用于多种压缩算法，显著提高压缩速度和效率。