少数层主导多数:揭示时间序列模型中的少层主导现象
The Few Govern the Many:Unveiling Few-Layer Dominance for Time Series Models
AI总结:
本文揭示时间序列模型中存在“缩放悖论”,即更大模型未必更好,并发现其根因是“少层主导”现象,进而提出自动识别并仅保留主导层的方法,在多个SOTA模型上以不到30%的层数在超95%任务中实现相当或更优精度。
摘要:
大规模模型处于时间序列(TS)预测的前沿,主要由两种范式主导:微调基于文本的大规模语言模型(LLM4TS)和从头训练时间序列基础模型(TSFMs)。两种方法都共享一个基本假设,即扩大模型容量和数据规模会带来性能提升。然而,我们在时间序列模型中观察到一个“缩放悖论”,揭示了一个令人困惑的现象:更大的模型并不能取得更好的性能。通过在两个模型家族、四个规模(1亿到17亿参数)和多样化数据(多达60亿观测值)上的大量实验,我们严格证实缩放悖论是一个普遍存在的问题。随后,我们通过分析内部表示来诊断其根本原因,识别出一种我们称之为“少层主导”的现象:只有一小部分层在功能上重要,而大多数层是冗余的、利用不足的,甚至可能干扰训练。基于这一发现,我们提出了一种实用方法,可以自动识别并仅保留这些主导层。在我们的模型中,仅保留21%的参数即可实现高达12%的精度提升和2.7倍的推理加速。我们在8个突出的SOTA模型(LLM4TS和TSFMs,9000万到60亿参数)上验证了该方法的普适性,结果表明在超过95%的任务中,保留不到30%的层即可达到相当或更优的精度。
英文摘要:
Large-scale models are at the forefront of time series (TS) forecasting, dominated by two paradigms: fine-tuning text-based Large Language Models (LLM4TS) and training Time Series Foundation Models (TSFMs) from scratch. Both approaches share a foundational assumption that scaling up model capacity and data volume leads to improved performance. However, we observe a \textit{\textbf{scaling paradox}} in TS models, revealing a puzzling phenomenon that larger models do \emph{NOT} achieve better performance. Through extensive experiments on two model families across four scales (100M to 1.7B parameters) and diverse data (up to 6B observations), we rigorously confirm that the scaling paradox is a pervasive issue. We then diagnose its root cause by analyzing internal representations, identifying a phenomenon we call \textit{few-layer dominance}: only a small subset of layers are functionally important, while the majority are redundant, under-utilized, and can even distract training. Based on this discovery, we propose a practical method to automatically identify and retain only these dominant layers. In our models, retaining only 21\% of the parameters achieves up to a 12\% accuracy improvement and a 2.7$\times$ inference speedup. We validate the universality of our method on 8 prominent SOTA models (LLM4TS and TSFMs, 90M to 6B), showing that retaining less than 30\% of layers achieves comparable or superior accuracy in over 95\% of tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。