在FMA协调的专家混合系统中的成本惩罚适应性:领域适应中分子记忆的实验证据
Cost-Penalized Fitness in FMA-Orchestrated Mixture of Experts: Experimental Evidence for Molecular Memory in Domain Adaptation
AI总结:
本文通过纳米FMT实验探讨了在数据分布变化时,MoE系统如何通过多样化而非替换积累领域专业知识,展示了分子记忆效应带来的高效领域迁移能力。
摘要:
我们通过七次受控运行展示了纳米FMT的实验结果,这是一种由自由市场算法(FMA)协调的变压器,具有动态专家混合(MoE)管理。实验探讨了在数据分布变化时,MoE系统如何管理其专家池的问题。我们证明,结合对新专家的线性宽容期的成本惩罚适应性指标,可以产生一个通过多样化而非替换积累领域专业知识的系统。核心结果是一个往返领域转移实验,显示返回先前学习领域时恢复速度提高9-11倍,无需专家出生或替换。这种“分子记忆”效应——即处于休眠状态的专家在领域返回时重新激活——在当前的MoE管理方法中没有先例。初步的成本分析估计,在中等场景下,一个OpenAI规模的提供商每年可节省3910万美元和271 GWh的能源减少。
英文摘要:
We present experimental results from seven controlled runs of nanoFMT, a Free-Market Algorithm (FMA) orchestrated transformer with dynamic Mixture-of-Experts (MoE) management. The experiments address a fundamental question for advanced LLM development: how should an MoE system manage its expert pool when operating at full capacity under changing data distributions? We demonstrate that cost-penalized fitness metrics, combined with a linear grace period for newborn experts, produce a system that accumulates domain expertise through diversification rather than replacement. The central result is a round-trip domain shift experiment showing 9-11x faster recovery when returning to a previously learned domain, with zero expert births or replacements required. This "molecular memory" effect -- where dormant experts survive and reactivate when their domain returns -- has no analogue in current MoE management approaches. A preliminary cost analysis estimates annual savings of $39.1M and 27.1 GWh energy reduction for an OpenAI-scale provider under a moderate scenario.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。