Recent years have witnessed the rapid advancements of large language models (LLMs) and their expanding applications, leading to soaring demands for computational resources. The widespread adoption of test-time scaling further intensifies the tension between model capability and resource consumption. However, a rigorous metric that accurately reflects an LLM's inference efficiency across diverse tokenizers, parameter counts, and model architectures remains absent. Motivated by the correlation between compression and intelligence, we introduce information capacity, a measure of model efficiency based on text compression performance relative to computational complexity. A distinctive feature of information capacity is its incorporation of tokenizer efficiency, which affects inference costs but is often neglected in LLM evaluations. We assess the information capacity of 56 open-source models and observe a consistent information capacity among different-sized models within a series. Experiments on five heterogeneous datasets reveal strong linguistic biases in mainstream LLMs. Empirical results verify the accuracy of performance prediction across model sizes based on information capacity and show the correlation between information capacity and benchmark scores. This metric can be used to quantify improvements in inference efficiency and provide insights into better scaling performance for future LLM development.
Compression of Generative Pre-trained Language Models via Quantization
The University of Hong Kong（香港大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
生成式预训练语言模型（PLM）的规模不断增大，极大地增加了对模型压缩的需求。尽管有各种方法来压缩BERT或其变体，但很少有人尝试压缩生成PLM，并且潜在的困难仍然不清楚。在本文中，我们压缩生成PLM量化。我们发现，以前的量化方法失败的生成任务，由于\textit{homogeneous word embeddings}所造成的能力降低，和\textit{varied distribution of weights}。相应地，我们提出了一个令牌级的对比蒸馏学习可区分的词嵌入，和一个模块的动态缩放，使量化器适应不同的模块。各种任务的实证结果表明，我们提出的方法优于国家的最先进的压缩方法生成PLM的一个明确的保证金。与全精度模型相比，我们在GPT-2和BART上分别实现了14.4倍和13.4倍的压缩率。
Text Compression for Sentiment Analysis via Evolutionary Algorithms
University of Cape Town（开普敦大学）；African Institute for Mathematical Sciences（非洲数学科学研究院）；South African Astronomical Observatory（南非天文台）
文本数据可以被智能地压缩而不损失评估情感的准确性吗？在本研究中，我们提出了一种新型的进化压缩算法PARSEC（PARts-of-Speech for sEntiment Compression），该算法利用词性标签来压缩文本，在与情感分析算法结合使用时，牺牲了最小的分类准确性。对12个英语情感数据集的八种商业和非商业情感分析算法的PARSEC的分析揭示，使用LingPipe（最准确的情感算法）的PARSEC的数据压缩的（20%，50%，75%）情感分类准确度损失（0%，1.3%，3.3%）的精确压缩是可能的。其他情感分析算法受到压缩的影响更严重。我们的结论是，显着的压缩文本数据是可能的情感分析取决于特定的应用程序和特定的情感分析算法使用的准确性要求。
