对序列数据中的长程依赖进行建模是序列学习的关键步骤。最近开发的模型Structured State Space (S4)在长程序列建模中展现出显著效果。然而,S4的成功究竟归因于其复杂的参数化和HiPPO初始化,还是仅仅因为State Space Models (SSMs),目前尚不清楚。为了进一步探究深度SSMs的潜力,我们从指数平滑(ETS)这一简单的SSM出发,通过将其直接融入逐元素MLP,提出了一种堆叠架构。我们为简单的ETS增加了额外参数和复数域,以降低归纳偏置。尽管增加的参数量不到逐元素MLP的1%,我们的模型在LRA基准上取得了与S4相当的结果。
英文摘要:
Modeling long-range dependencies in sequential data is a crucial step in sequence learning. A recently developed model, the Structured State Space (S4), demonstrated significant effectiveness in modeling long-range sequences. However, It is unclear whether the success of S4 can be attributed to its intricate parameterization and HiPPO initialization or simply due to State Space Models (SSMs). To further investigate the potential of the deep SSMs, we start with exponential smoothing (ETS), a simple SSM, and propose a stacked architecture by directly incorporating it into an element-wise MLP. We augment simple ETS with additional parameters and complex field to reduce the inductive bias. Despite increasing less than 1\% of parameters of element-wise MLP, our models achieve comparable results to S4 on the LRA benchmark.
MLP Fusion: Towards Efficient Fine-tuning of Dense and Mixture-of-Experts Language Models
School of Information Sciences, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校信息科学学院);Hong Kong Baptist University(香港浸会大学);Jacobs Technion-Cornell Institute, Cornell Tech(康奈尔理工学院雅各布斯以色列理工学院)
Incorporating Arbitrary Matrix Group Equivariance into KANs
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University(人工智能基础理论国家重点实验室,智能科学与技术学院,北京大学);Institute for Artificial Intelligence, Peking University(人工智能研究院,北京大学);Pazhou Laboratory (Huangpu), Guangzhou, Guangdong, China(黄埔实验室(琶洲),广州,广东,中国)
Geometric and Stochastic Analysis of Discontinuities in Sparse Mixture-of-Experts
Department of Mathematics, National University of Singapore, Singapore(新加坡国立大学数学系);Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology (HCMUT), VNU-HCM, Ho Chi Minh City, Vietnam(胡志明市技术大学计算机科学与工程学院)
fETSmcs: Feature-based ETS model component selection
Beihang University(北京航空航天大学);University of Manchester(曼彻斯特大学)
ETS(ExponenTial Smoothing or Error,Trend,Seasonality,指数平滑或误差、趋势、季节性)方法在状态空间表示中引入了一系列指数平滑模型,已被广泛用于自动预测。现有的ETS方法使用信息标准进行模型选择,通过在拟合到给定时间序列的所有模型中选择具有最小信息标准的最优模型。当应用于大规模时间序列数据时,在这种模型选择方案下的ETS方法遭受计算复杂性。为了解决这个问题,我们提出了一个有效的方法ETS模型选择训练分类器的模拟数据预测适当的模型组件的形式为一个给定的时间序列。我们提供了一个模拟研究,以显示所提出的方法对模拟数据的模型选择能力。我们评估我们的方法上广泛使用的预测竞争数据集M4,在点预测和预测区间。为了证明我们的方法的实用价值,我们展示了我们的方法在每月医院数据集上的性能改进。
Search to Capture Long-range Dependency with Stacking GNNs for Graph Classification
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Lenovo(联想);Paradigm. Inc(第四范式公司);Tsinghua University(清华大学)