arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.19201cs.CLcs.AIcs.LG
收藏

基于隐藏思维的高效推理

Efficient Reasoning with Hidden Thinking

Zhejiang University(浙江大学);Adobe(奥多比公司);Duke University(杜克大学);Oregon State University(俄勒冈州立大学);Northeastern University(东北大学)

Xuan Shen, Yizhou Wang, Xiangxi Shi, Yanzhi Wang, Pu Zhao, Jiuxiang Gu

AI总结:

提出高效推理框架Heima,通过Encoder将CoT压缩为隐藏表示以减少token用量,并利用Decoder重构推理过程,在提升效率的同时保持或提升零样本准确率。

摘要:

思维链(CoT)推理已成为提升多模态大语言模型(MLLMs)复杂问题解决能力的强大框架。然而,文本推理的冗长性带来了显著的低效问题。在本研究中,我们提出了Heima(即hidden llama),这是一个在隐藏潜空间中利用CoT推理的高效推理框架。我们设计了Heima Encoder,使用单个思维token将每个中间CoT压缩为紧凑的、更高层次的隐藏表示,有效减少了冗长性并降低了推理过程中所需的总体token数量。同时,我们设计了与传统的LLMs配合的Heima Decoder,将隐藏表示自适应地解释为可变长度的文本序列,重构出与原始CoT高度相似的推理过程。在多种推理MLLM基准上的实验结果表明,Heima模型在保持甚至提升零样本任务准确率的同时,实现了更高的生成效率。此外,使用Heima Decoder对多模态推理过程的有效重构,验证了我们方法的鲁棒性和可解释性。

英文摘要:

Chain-of-Thought (CoT) reasoning has become a powerful framework for improving complex problem-solving capabilities in Multimodal Large Language Models (MLLMs). However, the verbose nature of textual reasoning introduces significant inefficiencies. In this work, we propose $\textbf{Heima}$ (as hidden llama), an efficient reasoning framework that leverages reasoning CoTs at hidden latent space. We design the Heima Encoder to condense each intermediate CoT into a compact, higher-level hidden representation using a single thinking token, effectively minimizing verbosity and reducing the overall number of tokens required during the reasoning process. Meanwhile, we design corresponding Heima Decoder with traditional Large Language Models (LLMs) to adaptively interpret the hidden representations into variable-length textual sequence, reconstructing reasoning processes that closely resemble the original CoTs. Experimental results across diverse reasoning MLLM benchmarks demonstrate that Heima model achieves higher generation efficiency while maintaining or even better zero-shot task accuracy. Moreover, the effective reconstruction of multimodal reasoning processes with Heima Decoder validates both the robustness and interpretability of our approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑