Hecto：用于自适应与可解释推理的模块化稀疏专家模型
Hecto: Modular Sparse Experts for Adaptive and Interpretable Reasoning
AI总结：
Hecto 提出一种轻量级异构 MoE 架构，通过 Top-1 门控结合 GRU 时序专家与 FFNN 静态专家，在多个推理与回归任务上以更低资源实现与同构模型相当的性能，并验证了架构多样性对专业化、稳定性与可解释性的关键作用。
摘要：
专家混合（Mixture-of-Experts，MoE）模型通过将输入路由到专门的专家来实现条件计算，但这些专家依赖相同的归纳偏置，从而限制了表征多样性。这种静态计算路径对于需要不同类型推理的输入而言效率低下，并限制了专业化与可解释性。我们提出 Hecto，一种轻量级 MoE 架构，通过在稀疏 Top-1 门控机制下结合一个用于时序推理的 GRU 专家和一个用于静态抽象的 FFNN 专家来利用架构异构性。在三个推理基准（AG News、SST-2、HotpotQA）和一个回归任务（STS-B）上评估，尽管接收的是隔离的输入表征，Hecto 在性能上与同构基线持平或接近，同时实现了清晰的专家专业化，每个专家对应不同的推理类型（时序 vs 静态）。在更大的批大小下，Hecto 表现出更好的性能，受益于放宽的计算约束使其异构架构能够更有效地优化。消融结果将架构多样性确定为 Hecto 在多样化推理任务中稳定性和可解释性的来源。总体而言，Hecto 确立了自身作为条件计算新基准的地位，为低资源场景下的专业化推理提供了一个原则性框架，其模型优势来源于原则性的专业化。
英文摘要：
Mixture-of-Experts (MoE) models enable conditional computation by routing inputs to specialized experts, but these experts rely on identical inductive biases, thus limiting representational diversity. This static computation pathway is inefficient for inputs that require different types of reasoning and limits specialization and interpretability. We propose Hecto, a lightweight MoE architecture that leverages architectural heterogeneity by combining a GRU expert for temporal reasoning and an FFNN expert for static abstraction under a sparse Top-1 gating mechanism. Evaluated on three reasoning benchmarks (AG News, SST-2, HotpotQA) and a regression task (STS-B), Hecto matches or closely trails homogeneous baselines in performance despite receiving isolated input representations, while achieving clear expert specialization, with each expert aligning to distinct reasoning types (temporal vs static). At larger batch sizes, Hecto exhibits improved performance, benefiting from relaxed computational constraints that allow its heterogeneous architecture to optimize more effectively. Ablation results isolate architectural diversity as the source of Hecto's stability and interpretability across diverse reasoning tasks. Overall, Hecto establishes itself as a new benchmark for conditional computation, offering a principled framework for specialized reasoning in low-resource regimes with its model strength derived from principled specialization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
