CosmicFish-HRM：紧凑语言模型中基于层次循环机制的适应性推理
CosmicFish-HRM: Adaptive Reasoning via Hierarchical Recurrent Mechanisms in Compact Language Models
AI总结：
提出一种紧凑语言模型CosmicFish-HRM，通过层次推理模块动态分配推理深度，在保持较小参数量的同时实现适应性推理。
摘要：
大型语言模型已经实现了强大的推理能力，尽管通常以巨大的参数数量和昂贵的推理为代价。在这项工作中，我们探索了一个不同的方向：紧凑语言模型中的自适应推理深度。我们提出了CosmicFish-HRM，这是一个紧凑的语言模型，围绕一个层次推理模块（HRM）构建，该模块在推理过程中动态分配计算资源。该模型不是对每个输入应用固定的计算，而是迭代通过高层和低层推理循环，并根据输入复杂度学习何时停止。CosmicFish-HRM将这种自适应推理核心与现代Transformer组件（包括分组查询注意力、RoPE和SwiGLU激活）相结合。虽然额外的推理基础设施在小规模下引入了开销，但我们假设随着模型规模的增长和HRM核心相对成本的降低，这种权衡变得越来越有利。我们的结果表明，该模型学习了非均匀的推理行为，在不同任务和输入之间分配不同数量的推理步骤。这些发现表明，自适应推理深度可能为仅依赖参数规模来实现推理能力提供一种有前途的替代方案。
英文摘要：
Large language models have achieved strong reasoning capabilities, though often at the cost of massive parameter counts and expensive inference. In this work, we explore a different direction: adaptive reasoning depth in compact language models. We present CosmicFish-HRM, a compact language model built around a Hierarchical Reasoning Module (HRM) that dynamically allocates computational effort during inference. Instead of applying fixed computation to every input, the model iterates through high-level and low-level reasoning cycles and learns when to halt based on input complexity. CosmicFish-HRM combines this adaptive reasoning core with modern transformer components including Grouped Query Attention, RoPE, and SwiGLU activations. While the additional reasoning infrastructure introduces overhead at small scale, we hypothesize that this tradeoff becomes increasingly favorable as model size grows and the relative cost of the HRM core diminishes. Our results show that the model learns non-uniform reasoning behavior, allocating different numbers of reasoning steps across tasks and inputs. These findings suggest that adaptive reasoning depth may offer a promising alternative to relying solely on parameter scale for reasoning capability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
