In order to address the chain of thought in the large language model inference cost surge, this research proposes to use a sparse attention mechanism that only focuses on a few relevant tokens. The researcher constructed a new attention mechanism and used GiantRabbit trained with custom GPTs as an experimental tool. The experiment tested and compared the reasoning time, correctness score and chain of thought length of this model and o1 Preview in solving the linear algebra test questions of MIT OpenCourseWare. The results show that GiantRabbit's reasoning time and chain of thought length are significantly lower than o1 Preview. It verifies the feasibility of sparse attention mechanism for optimizing chain of thought reasoning. Detailed architectural details and experimental process have been uploaded to Github, the link is:https://github.com/brucewang123456789/GeniusTrail.git.
Tracing the Thought of a Grandmaster-level Chess-Playing Transformer
Shanghai Innovation Institute, Shanghai, China(上海创新研究院);School of Computer Science, Fudan University, Shanghai, China(复旦大学计算机科学技术学院);School of Mathematics and Statistics, Xi’an Jiaotong University(西安交通大学数学与统计学院)
Spectral-Progressive Thought Flow for Lightweight Multimodal Reasoning
Informatics Institute, University of Amsterdam, Amsterdam, The Netherlands(阿姆斯特丹大学信息学院);Department of Computer Science, University College London(伦敦大学学院计算机科学系);University of Thessaly, Volos, Greece(塞萨洛尼基大学);Department of Electronic and Electrical Engineering, Trinity College Dublin, Dublin, Ireland(都柏林信任学院电子与电气工程系);School of Information Technology, Halmstad University, Halmstad, Sweden(哈姆斯塔德大学信息科技学院);Amazon AGI, Seattle, USA(亚马逊人工智能研究部)
Retrieval-of-Thought: Efficient Reasoning via Reusing Thoughts
Department of Computer Science and Engineering, University of Minnesota(明尼苏达大学计算机科学与工程系);Department of Computer Science and Engineering, University of California Santa Cruz(加州大学圣克鲁兹分校计算机科学与工程系);Argonne National Laboratory(阿贡国家实验室)
Sparse Phased Array Optimization Using Deep Learning
Department of Computer Science, University of California, Irvine(加州大学尔湾分校计算机科学系);School of Electrical and Computer Engineering, Tel Aviv University(特拉维夫大学电气与计算机工程学院)
A*-Thought-V2: Efficient Latent Reasoning via Geometric Dynamics of LLM
Beijing University of Posts and Telecommunications(北京邮电大学);The Hong Kong Polytechnic University(香港理工大学);Tsinghua University(清华大学);JIUTIAN Research(中移九天);OpenBMB
Beyond Dense States: Sparse Transcoders as Causally Testable Operators for LLM Latent Reasoning
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics(南京航空航天大学计算机科学与技术学院);MIIT Key Laboratory of Pattern Analysis and Machine Intelligence(信息产业部模式分析与机器智能重点实验室);Institute for AI, Tsinghua University, Beijing, China(清华大学人工智能研究院)
Beyond the Commitment Boundary: Probing Epiphenomenal Chain-of-Thought in Large Reasoning Models
CLCG, University of Groningen(格罗宁根大学CLCG);University of Milano-Bicocca(米兰-布雷拉大学);University of Trieste(特里耶大学);Khoury College of Computer Sciences, Northeastern University(东北大学Khoury计算机科学学院)