The Sentence-State LSTM (S-LSTM) is a powerful and high efficient graph recurrent network, which views words as nodes and performs layer-wise recurrent steps between them simultaneously. Despite its successes on text representations, the S-LSTM still suffers from two drawbacks. Firstly, given a sentence, certain words are usually more ambiguous than others, and thus more computation steps need to be taken for these difficult words and vice versa. However, the S-LSTM takes fixed computation steps for all words, irrespective of their hardness. The secondary one comes from the lack of sequential information (e.g., word order) that is inherently important for natural language. In this paper, we try to address these issues and propose a depth-adaptive mechanism for the S-LSTM, which allows the model to learn how many computational steps to conduct for different words as required. In addition, we integrate an extra RNN layer to inject sequential information, which also serves as an input feature for the decision of adaptive depths. Results on the classic text classification task (24 datasets in various sizes and domains) show that our model brings significant improvements against the conventional S-LSTM and other high-performance models (e.g., the Transformer), meanwhile achieving a good accuracy-speed trade off.
Pattern Recognition Center, WeChat AI, Tencent Inc（腾讯公司微信人工智能模式识别中心）
Depth-adaptive neural networks can dynamically adjust depths according to the hardness of input words, and thus improve efficiency. The main challenge is how to measure such hardness and decide the required depths (i.e., layers) to conduct. Previous works generally build a halting unit to decide whether the computation should continue or stop at each layer. As there is no specific supervision of depth selection, the halting unit may be under-optimized and inaccurate, which results in suboptimal and unstable performance when modeling sentences. In this paper, we get rid of the halting unit and …
Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves
Aleph Alpha Research, Heidelberg, Germany（Aleph Alpha研究机构）；Technical University of Munich, Germany（慕尼黑技术大学）；Research Group Social Computing, Technical University of Munich, Germany（社会计算研究组，慕尼黑技术大学）；Lab1141, Germany（Lab1141研究机构）；Computer Science Department, TU Darmstadt, Germany（图林根达姆施塔特技术大学计算机科学系）
KingSoft AI Lab（金山AI实验室）；University of Groningen（格罗宁根大学）
Learning deeper models is usually a simple and effective approach to improve model performance, but deeper models have larger model parameters and are more difficult to train. To get a deeper model, simply stacking more layers of the model seems to work well, but previous works have claimed that it cannot benefit the model. We propose to train a deeper model with recurrent mechanism, which loops the encoder and decoder blocks of Transformer in the depth direction. To address the increasing of model parameters, we choose to share parameters in different recursive moments. We conduct our experim…
While recurrent models have been effective in NLP tasks, their performance on context-free languages (CFLs) has been found to be quite weak. Given that CFLs are believed to capture important phenomena such as hierarchical structure in natural languages, this discrepancy in performance calls for an explanation. We study the performance of recurrent models on Dyck-n languages, a particularly important and well-studied class of CFLs. We find that while recurrent models generalize nearly perfectly if the lengths of the training and test strings are from the same range, they perform poorly if the t…
Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
ELLIS Institute Tübingen（埃利斯研究所蒂宾根分所）；Max-Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Tübingen AI Center（蒂宾根人工智能中心）；University of Maryland, College Park（马里兰大学帕克分校）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
University of Maryland（马里兰大学）；New York University（纽约大学）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；University of North Carolina（北卡罗来纳大学）；ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center（图宾根ELLIS研究所、马克斯·普朗克智能系统研究所、图宾根人工智能中心）；Columbia University（哥伦比亚大学）
ETH Zürich（苏黎世联邦理工学院）；The Swiss AI Lab IDSIA (USI-SUPSI)（瑞士人工智能实验室IDSIA（卢加诺大学-瑞士南部应用科学与艺术大学））；NNAISENSE
Many sequential processing tasks require complex nonlinear transition functions from one step to the next. However, recurrent neural networks with 'deep' transition functions remain difficult to train, even when using Long Short-Term Memory (LSTM) networks. We introduce a novel theoretical analysis of recurrent networks based on Gersgorin's circle theorem that illuminates several modeling and optimization issues and improves our understanding of the LSTM cell. Based on this analysis we propose Recurrent Highway Networks, which extend the LSTM architecture to allow step-to-step transition depth…
The ALTA Institute（ALTA研究所）；Computer Laboratory（计算机实验室）；University of Cambridge（剑桥大学）
We investigate an extension of continuous online learning in recurrent neural network language models. The model keeps a separate vector representation of the current unit of text being processed and adaptively adjusts it after each prediction. The initial experiments give promising results, indicating that the method is able to increase language modelling accuracy, while also decreasing the parameters needed to store the model along with the computation required at each step.
Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching
Technical University of Munich (TUM)（慕尼黑工业大学（TUM））；Imperial College London（帝国理工学院）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；University of Potsdam（波茨坦大学）；Hasso Plattner Institute for Digital Engineering（哈索·普拉特纳数字工程研究所）
