This paper investigates the gap in representation powers of Recurrent Neural Networks (RNNs) and Transformers in the context of solving algorithmic problems. We focus on understanding whether RNNs, known for their memory efficiency in handling long sequences, can match the performance of Transformers, particularly when enhanced with Chain-of-Thought (CoT) prompting. Our theoretical analysis reveals that CoT improves RNNs but is insufficient to close the gap with Transformers. A key bottleneck lies in the inability of RNNs to perfectly retrieve information from the context, even with CoT: for several tasks that explicitly or implicitly require this capability, such as associative recall and determining if a graph is a tree, we prove that RNNs are not expressive enough to solve the tasks while Transformers can solve them with ease. Conversely, we prove that adopting techniques to enhance the in-context retrieval capability of RNNs, including Retrieval-Augmented Generation (RAG) and adding a single Transformer layer, can elevate RNNs to be capable of solving all polynomial-time solvable problems with CoT, hence closing the representation gap with Transformers.
Pre-training with Large Language Model-based Document Expansion for Dense Passage Retrieval
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Kuaishou Technology（快手科技）
VimoRAG: Video-based Retrieval-augmented 3D Motion Generation for Motion Language Models
Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；University of Macau（澳门大学）；University of Surrey（Surrey大学）；Nanjing University（南京大学）；Nanyang Technological University（南洋理工大学）；National University of Singapore（新加坡国立大学）
Incorporating Retrieval-based Causal Learning with Information Bottlenecks for Interpretable Graph Neural Networks
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Shanghai Jiao Tong University（上海交通大学）；Key Laboratory of Machine Intelligence and Advanced Computing, Sun Yat-sen University（中山大学机器智能与先进计算重点实验室）
Attention is All You Need? Good Embeddings with Statistics are enough:Large Scale Audio Understanding without Transformers/ Convolutions/ BERTs/ Mixers/ Attention/ RNNs or ....
CoT-MAE v2: Contextual Masked Auto-Encoder with Multi-view Modeling for Passage Retrieval
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Kuaishou Technology（快手科技）
ASK: Adaptive Self-improving Knowledge Framework for Audio Text Retrieval
University of Electronic Science and Technology of China（电子科技大学）；Hong Kong University of Science and Technology（香港科技大学）；Mohamed Bin Zayed University of Artificial Intelligence（莫扎德人工智能大学）；Peking University（北京大学）；Tsinghua University（清华大学）
