arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.13624cs.CL
收藏

超越统一令牌:时间序列语言模型的自适应压缩

Beyond Uniform Tokens: Adaptive Compression for Time Series Language Models

Zhejiang University(浙江大学);Harbin Institute of Technology(哈尔滨工业大学);Shandong University(山东大学)

Jialin Gan, Xin Qiu, Guangzhe Chen, Xue Wang

AI总结:

针对时间序列语言模型中令牌效率低的问题,提出自适应令牌预算框架,通过频域结构压缩时间序列令牌并逐层减少提示令牌,实现高达7.68倍推理加速并在78%设置中提升性能。

摘要:

大型语言模型(LLM)通过共享令牌接口联合建模数值观测和文本上下文,实现了时间序列(TS)分析。然而,TS令牌和提示令牌表现出根本不同的信息结构,使得统一令牌处理效率低下。在本文中,我们从非对称令牌的角度研究TS语言建模中的令牌效率。我们表明,TS令牌具有高度不均匀的频谱贡献,其中许多令牌共享冗余频率模式,而一小部分保留了关键的时间证据。我们还观察到,提示令牌的影响随模型深度衰减,表明在所有层中完全保留提示是不必要的。基于这些发现,我们开发了一个自适应令牌预算框架,通过频域结构压缩TS令牌,并逐层减少提示令牌。在预测、分类、插补和异常检测上的实验表明,在\textit{\textbf{78\%}}的评估设置中实现了高达\textit{\textbf{7.68$\times$}}的推理加速和性能提升,显示了非对称令牌压缩对于可扩展TS基础模型的有效性。

英文摘要:

Large language models (LLMs) have enabled time series (TS) analysis by jointly modeling numerical observations and textual context through a shared token interface. However, TS tokens and prompt tokens exhibit fundamentally different information structures, making uniform token processing inefficient. In this paper, we study token efficiency in TS language modeling from an asymmetric-token perspective. We show that TS tokens have highly uneven spectral contributions, where many tokens share redundant frequency patterns while a small subset preserves critical temporal evidence. We also observe that prompt-token influence attenuates with model depth, suggesting that full prompt retention across all layers is unnecessary. Based on these findings, we develop an adaptive token budgeting framework that compresses TS tokens via frequency-domain structure and progressively reduces prompt tokens across layers. Experiments across forecasting, classification, imputation, and anomaly detection demonstrate up to \textit{\textbf{7.68$\times$}} inference acceleration and performance gains in \textit{\textbf{78\%}} of evaluated settings, showing the effectiveness of asymmetric token compression for scalable TS foundation models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑